Server Quality Checklist

Tool Scores

• list_experimentsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the disclosure burden. It reveals ordering and implies that destroyed experiments are still listed, but it does not explicitly state that this is a read-only operation or describe default filter behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two crisp sentences with no filler. The core action and ordering are front-loaded, and the use-case guidance earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Complete schema coverage, no required parameters, and an output schema make this nearly complete. The description adds practical selection context, though it could explicitly note that listing is side-effect-free.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the description adds no parameter-specific meaning beyond the schema. Since the schema fully documents limit and status, a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('List') and resource ('experiments'), with an ordering guarantee ('most recent first'). It clearly differentiates from singular get_experiment by plural scope, but does not explicitly name sibling alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides concrete use cases: finding a previously created sandbox and checking for undeleted sandboxes before creating another. This gives clear context for when to use the tool, though it does not mention exclusions or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_experimentA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries the full disclosure burden and meets it by enumerating the returned contents: status, base image, isolation settings, resource limits, command counts, last test summary, change statistics and artifact count. This is unusually transparent for a read tool, effectively previewing the output schema, though it does not address error or edge-case behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is organized into three scannable sections — a one-line purpose statement, a 'USE THIS' directive, and a 'RETURNS' list — with the most decision-relevant guidance front-loaded. No sentence is wasted, though the RETURN enumeration is slightly verbose for a tool that already declares an output schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-parameter read tool, the description covers purpose, usage triggers, and return contents comprehensively, and an output schema is declared so return values are further specified structurally. Nothing an agent needs in order to call it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter experiment_id is 100% covered by the schema description ('The experiment to describe.'), so the schema already carries the meaning. The description adds nothing parameter-specific beyond the schema, holding it at the baseline 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Fetch') and resource ('an experiment's full state and a summary of what happened in it'), clearly scoping the tool's function. It is distinguishable from siblings like get_job_status and check_sandbox_runtime by the breadth of state described, though it does not explicitly name a sibling to differentiate against, so it misses full marks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes an explicit 'USE THIS to re-orient' directive with concrete trigger conditions: after a long gap, or to check sandbox liveness before sending more commands. This gives clear when-to-use context, though it stops short of naming alternatives or stating when NOT to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_sandbox_fileA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that paths are workspace-relative and cannot escape the sandbox, and that it reads as text. However, it does not mention behavior for missing files, binary files, or permission errors. It provides some behavioral context but not exhaustive detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose, then the usage context, then the key constraint. Every sentence earns its place; there is no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read tool with an output schema present, the description covers the essential context: what it reads, when to use it, and a critical limitation. It doesn't explain error handling, but that's minor given the tool's simplicity and the existence of an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so both parameters are already described. The description adds the note that paths are workspace-relative, which reinforces the path parameter's meaning, but it does not add new semantics beyond what the schema provides. Baseline 3 applies given high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads a file from inside the sandbox as text, using a specific verb and resource. It distinguishes itself from sibling tools like write_sandbox_file by focusing on reading, and explicitly notes the sandbox confinement, making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides an explicit use case: 'USE THIS to investigate a failure -- read the source, the config, a log'. It does not explicitly mention when not to use it or name alternatives, but the guidance is clear enough for an agent to select it for file inspection. It also clarifies it cannot read the developer's machine, which sets expectations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• collect_artifactsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosing side effects. It clearly states that the tool copies files out of a sandbox (non-destructive on the source) and explicitly notes that it writes to the server's own state directory and cannot write to the developer's machine. It does not, however, describe the return value or any potential side effects like deleting the sandbox, so it is transparent but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise and well-structured. Three short sentences cover purpose, usage context, and pattern/limitation details without any redundancy or filler. Every sentence adds distinct value, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides sufficient context for an agent to invoke the tool correctly: what it does, when to use it, what parameters to pass (with format), and a key side-effect limitation. It does not describe the output format (though an output schema exists in the context), and it omits edge cases like no-matching-files behavior, so it is not fully complete but is adequate for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides 100% coverage with descriptions for both parameters, and the tool description supplements these with additional explanation of how patterns work ('Workspace-relative shell globs, e.g. ['dist/*.js'], or '**/name' for a recursive search'). This goes beyond the schema, offering practical clarification, so it earns a 4 rather than the baseline 3 for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Copy selected files out of a sandbox before it is destroyed') and specifies the resource (files) and scope (selected, via globs). It also provides a usage context ('USE THIS for build output, test reports, coverage, benchmark results or logs you want to keep or quote') that distinguishes its intent from pure file reading or writing. However, it does not explicitly name any sibling tool as an alternative, unlike the high-reference example, so it falls short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool ('USE THIS for build output, test reports, coverage, benchmark results or logs you want to keep or quote'), which gives clear guidance. It also clarifies the pattern format and a key limitation ('this tool cannot write anywhere on the developer's machine'). However, it does not explicitly state when not to use it or name alternative tools, so it is not fully explicit on all usage boundaries.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_jobA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses the destructive action (kill process) and adds safety: 'Safe on a job that already finished -- it returns the final state untouched.' This gives an idempotency guarantee. Could mention side effects on logs or output, but for a simple kill action this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no wasted words. The core purpose is front-loaded, and usage guidance is compact. Each sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, return values are covered. The description covers purpose, usage, and safety for a simple one-parameter cancellation tool. Could specify synchronous/asynchronous behavior, but not essential. Completeness is good for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and job_id is described as 'Job id to cancel.' The description adds no additional semantics beyond the schema (e.g., format, where to find it). Since coverage is high, baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb+resource: 'Stop a running command and kill its process inside the sandbox.' This clearly differentiates from siblings like get_job_status or get_job_result, which are read-only. The purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use context: 'USE THIS when a command is clearly stuck or no longer needed.' Also notes safety on already-finished jobs, giving practical guidance. Does not name alternatives explicitly but the context is sufficient to select this tool over read-only siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_statusA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full burden. It implies a non-mutating status check and mentions returned values, but it does not explicitly state side effects, idempotency, or authorization requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, with two short sentences that communicate purpose, output, and the follow-up action without unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple polling status tool, the description is nearly complete: it defines what is returned, the completion condition, and the next step. It lacks an explicit output schema or status value enumeration, but these are not critical given the simple use case.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single param job_id is described in the schema as 'Job id from execute_experiment,' which is clear. The tool description adds no further parameter detail, but schema coverage is complete for the only input.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: checking on a background command started with background=true, and it explicitly distinguishes its output (status and elapsed milliseconds) from the later retrieval step (get_job_result).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives direct usage guidance: poll this tool until 'finished' is true, then call get_job_result. This explicitly names the companion tool and the correct sequencing, leaving little ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• destroy_experimentA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the destructive nature ('destroy everything'), idempotency (safe on already-destroyed), and return behavior (final report). This adequately informs an agent of side effects and safety, though it does not mention reversibility or permissions, which are implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the primary action, then a clear usage directive, idempotency note, and return info. Each sentence adds value with no fluff, and the capitalized section labels make it easily scannable for an agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, return format details are covered. The description provides purpose, usage trigger, idempotency, and return contents, which is fully sufficient for an agent to decide when and how to call the tool correctly. Nothing essential is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes the single parameter (experiment_id) at 100% coverage. The description adds no additional parameter-specific semantics beyond what the schema provides—only idempotency context that is behavioral, not parameter-related. Thus, baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action (destroy), the resource (a sandbox/experiment), and the effect ('everything in it'). It distinguishes from siblings like cancel_job and get_experiment by emphasizing final cleanup and returning a report, making its unique purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs 'USE THIS as soon as an experiment has told you what you needed' and emphasizes 'Always call it' to free CPU/memory, giving a clear trigger condition. It does not mention specific alternatives or when not to use it, but the context is clear execution is for final cleanup.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_resultA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the important blocking wait behavior and timeout bound, plus the exact return shape. It doesn't cover failure/timeout outcomes, but the core behavioral trait is clearly stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with zero waste. The primary purpose is front-loaded, and the return shape and blocking behavior are compactly conveyed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given one parameter, full schema coverage, and the presence of an output schema, the description covers the essential behavioral context: what it returns and when it returns. Nothing critical is missing for a straightforward fetch tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the single parameter job_id is described as 'Job id to fetch.' The description adds no additional meaning beyond the schema, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Fetch') and resource ('completed result of a background command'). It also references execute_experiment's return shape, which precisely defines what 'result' means and distinguishes it from the sibling get_job_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context: this fetches results and waits for completion, bounded by the job's timeout. It doesn't explicitly name alternatives or say when not to use it, but the waiting behavior implies get_job_status is for non-blocking status checks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_sandbox_runtimeA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It states it returns Docker version and isolation defaults, implying a read-only diagnostic operation, and does not contradict any annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with purpose, and includes usage guidance without any redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists (though not shown) and the tool has no parameters, the description provides sufficient context: when to use it and what it returns. It does not cover potential error scenarios, but they are likely minor for a diagnostic tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description adds no parameter details, but this is unnecessary. The baseline for no parameters is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool confirms sandbox runtime availability and reports active defaults, with a specific verb and resource. It distinguishes itself from siblings by explicitly tying its use to diagnosing create_experiment failures.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance ('USE THIS first if create_experiment fails') and the diagnostic purpose, but does not mention when not to use it or name specific alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_experimentsA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries the full burden — and it delivers. It discloses the exact return payload (test results, failing test names, exit codes, change size, duration, artifact counts, recommendation) and a non-obvious trait: destroyed experiments remain comparable because findings were recorded before teardown. This adds real value beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-organized with clear lead-in markers (USE THIS, RETURNS). The purpose is front-loaded, followed by usage guidance and return/behavioral notes. Every sentence earns its place with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists, so the return-value summary is a bonus rather than a necessity. The destroyed-experiments note covers a likely edge case, and the minItems constraint is echoed. Slightly more on comparison criteria between experiments could push this to 5, but nothing needed to call the tool correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with both parameters fully documented including the labels example. The description adds no new parameter-level detail beyond echoing the 'two or more' minItems constraint, so the baseline of 3 applies — the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb and resource ('Compare two or more experiments side by side') with a clear distinct scope from siblings. It is clearly a comparison/analysis tool, not a creation or execution one, so an agent can separate it from create_experiment, execute_experiment, and get_experiment without opening schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives concrete when-to-use scenarios ('three candidate fixes, two runtime versions... recommend one') and instructs to run each approach in its own experiment first. It lacks an explicit when-not-to-use clause or named alternatives, but the use-case is unambiguous enough that the agent can route correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_changesA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that build output and dependency directories are excluded, preventing noise. While it does not explicitly say the tool is read-only, the phrasing 'inspect' and 'before destroying a sandbox' strongly implies no side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections for purpose, usage, and return values. It is concise yet complete, using formatting like bold headings to highlight key points.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the output (file lists, counts, optional diff) and mentions exclusions, which is essential for an inspection tool. It lacks details on error handling, but the context and schema provide sufficient information for typical usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema descriptions for all three parameters are clear and adequate. The description only adds minimal context about the optional diff, which is already implied by the schema's 'include_diff' field.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool shows what the experiment changed relative to the original project. It distinguishes itself from siblings by focusing on inspecting changes rather than listing, running, or destroying experiments.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is given to use this tool before destroying a sandbox and before reporting findings to the developer. It also states what the tool returns, helping the agent know when to invoke it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_sandbox_fileA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description must carry the behavioral burden. It discloses that writes only land in the sandbox copy, that parent directories are created automatically, and that no tool writes to the developer's project. It could also mention overwriting behavior or error conditions, but the most critical safety and side-effect information is covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short paragraphs, each with a distinct purpose: core action, usage guidance, and safety note. The primary action is front-loaded, and every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a write tool with three required parameters and an existing output schema, the description covers the key use case, the safety boundary, and the recommended workflow. It tells the agent exactly when to use it and how to handle changes that should persist, leaving no critical gaps for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so each parameter is already documented. The description adds no parameter-specific semantics beyond what the schema provides, except implicitly through the behavior of creating parent directories. Per guidelines, a high-coverage schema warrants a baseline score of 3, and the description does not go beyond that.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the exact action (write a file inside the sandbox), the resource (sandbox), and the key behavior (creating parent directories). It also contrasts with the absence of any tool that writes to the developer's project, distinguishing it from the sibling set where no other write tool exists.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs when to use this tool ('USE THIS to apply a candidate fix') and when not to ('if a change is worth keeping, show them the diff from inspect_changes and let them apply it'). Also recommends it over shell heredocs, providing concrete selection criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_experimentA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden of disclosing behavior. It clearly states that a non-zero exit code is normal (not an error), returns stdout/stderr/duration, and explains that background mode returns a job id. It does not explicitly mention potential filesystem side effects, but that is inherent to running a shell command in a sandbox; the description is otherwise transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: one sentence states the core function, followed by usage guidance, return semantics, and background handling. It avoids redundant details and every sentence contributes useful information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's relative simplicity and the presence of an output schema (implied by the return description), the description provides sufficient context: it explains what happens on execution, how to handle background vs. foreground, and how to retrieve results using sibling tools. No critical information is missing for correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides descriptions for all five parameters (100% coverage), so the baseline is 3. The description adds value by clarifying the intent of background (for long-running tasks) and tying timeout to the sandbox's default, which goes beyond the schema's factual description. No parameter is left unexplained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a shell command inside a sandbox, which is specific and distinct from sibling tools that manage experiments or files. It also explicitly mentions the return values (exit code, stdout, stderr, duration) and background execution, leaving no ambiguity about its purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says to use this for terminal-like tasks (installs, builds, scripts, migrations, exploration) and contrasts it with host execution. It also provides clear guidance on when to set background=true and how to poll via get_job_status/get_job_result, plus timeout behavior.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_experimentA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully carries the transparency burden. It explains snapshot copying, lack of propagation, secret handling, network defaults, resource caps, mount strategies, and failure behavior in detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear, front-loaded opening sentences and purposeful paragraph breaks. It is detailed but every sentence adds meaningful information about behavior, safety, or usage, with no filler or tautology.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity, the description is remarkably complete. It covers the full lifecycle of creating an experiment, input parameter implications, safety mechanisms, return information, and failure behavior. The presence of an output schema means return values do not need to be exhaustively restated, but the description still usefully mentions experiment_id, warnings, and applied isolation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes every parameter with 100% coverage, so the baseline is 3. The description adds useful contextual semantics beyond the schema, such as recommending 'restricted' network for package installation, warning that writable host mounts do not exist, and noting that credential-shaped environment names are refused.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: create a disposable, isolated environment and copy a project into it. This distinguishes it from sibling tools like list_experiments, execute_experiment, and get_experiment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on when to use this tool, such as 'USE THIS when you are about to do something you should not do on the developer's machine' and provides concrete examples like installing dependencies or running builds. It also clarifies safe defaults and opt-in behaviors.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_testsA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Even though no annotations are provided, the description is highly transparent about what the tool does: it detects the runner by inspecting the sandbox, runs the suite, parses test counts, and returns exit code, stdout/stderr, duration, and a summary. It also adds a important caveat about trusting the exit code when test_summary.detected is false.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections: the core action, usage guidance, return values, and parameter override note. It is concise yet information-dense, with no redundant or vague phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete for a tool of this scope: it explains what runs, how detection works, what the output contains, and how to interpret edge cases. It also fits well alongside sibling tools like execute_experiment, making the tool's niche clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All three parameters are described in the schema, and the description adds useful semantic context beyond the schema. It explains the command parameter's role ('Pass command to override detection'), clarifies timeout as 'Seconds before the run is killed', and identifies experiment_id as the test target.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Run the project's test suite') and the context ('inside a sandbox'), with the explicit goal of parsing results. It also distinguishes this tool from execute_experiment by specifying that it is for checking whether the project still works.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on when to use this tool ('USE THIS instead of execute_experiment when you want to know whether the project still works') and explains the auto-detection behavior. It also tells the user how to override detection with the command parameter.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.