Server Quality Checklist

Tool Scores

• checkpoint_environmentA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate a mutation (readOnlyHint=false) and non-destructive (destructiveHint=false). The description adds behavioral nuance by stating 'The original is untouched' and that the fork costs a delta rather than a rebuild, which explains the side-effect profile beyond the annotations. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, using two sentences to convey purpose, behavior, and usage rationale. It is front-loaded with the primary action and avoids unnecessary filler, earning each sentence's place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema, the description should clarify what the tool returns or produces, but it does not mention the result (e.g., new environment ID). It also omits prerequisites like the existence of a sealed snapshot. While it covers the cost model and non-destructive nature, it lacks sufficient detail for an agent to know what to expect beyond a side effect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50% since 'environment' lacks a description. The description does not explain the parameters at all; it never clarifies that 'environment' refers to the source identifier or that 'name' is for the new fork. It fails to compensate for the missing parameter documentation, leaving the agent without essential semantic guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (fork) and resource (environment) with specific scope: 'Fork an environment from its last sealed snapshot into a new one.' It distinguishes from siblings like create_environment by emphasizing the fork originates from a snapshot and leaves the original untouched, which is a clear differentiator.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance on when to use the tool: 'This is how you try several approaches from one prepared state.' It conveys the intended use case, though it does not mention when not to use it or contrast with create_environment directly. Still, the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_environmentA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds meaningful behavioral context beyond the annotations: persistence across calls, optional git cloning, and the return value. This complements annotations like openWorldHint=true (cloning) and readOnlyHint=false (creation). It does not mention resource/credit implications or failure modes, but covers the core behavioral contract well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the essential function, and no filler. It efficiently communicates persistence, optional cloning, and the return id—every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential facts for a creation tool with no output schema: what is created, persistence, optional repo cloning, and the return id. It does not discuss edge cases like clone failures, name conflicts, or costs, but given the moderate complexity and strong schema/annotations, it is sufficiently complete for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all three parameters (ref, name, repo) are already documented structurally. The description's mention of optional repository cloning adds a small layer of meaning, but it does not enrich parameter semantics beyond what the schema provides, matching the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Create') and clearly identifies the resource ('persistent development environment') with defining traits: a filesystem and processes that survive between calls. It also distinguishes from siblings by noting the returned environment id is used by every other tool, making the tool's role as the foundational creation step unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool by stating it returns the environment id used by every other tool, establishing it as the prerequisite step. However, it does not explicitly name alternatives or state when not to use it (e.g., prefer run_command for existing environments), so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_commandA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate openWorldHint=true and destructiveHint=true, but the description adds valuable behavioral context: it is not a shell, it returns exit code and output, and a paused environment resumes for it. These details go beyond the annotations and help the agent understand execution semantics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose. Each sentence adds essential information without redundancy, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 parameters with nested objects and no output schema, the description covers the primary behavior (command execution, exit code/output) and key usage notes (argv, shell). It lacks detailed parameter explanations but the schema covers timeout_ms, and the annotations cover safety aspects. This is reasonably complete for a command runner tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at only 20%, the description should compensate. It adds meaning to the argv parameter (list form, explicit shell usage) and mentions environment behavior, but does not explain cwd, env, or environment beyond that. The timeout_ms parameter already has a schema description. This is adequate but not comprehensive.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Run one command in an environment and get its exit code and output,' which clearly identifies the verb, resource, and outcome. It also distinguishes itself from a shell, clarifying its scope and differentiating it from sibling tools that manage environments.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage instructions: 'pass argv as a list, and ask for a shell explicitly with ["/bin/sh","-lc","…"] if you want one.' It also notes that a paused environment resumes to serve the command, giving practical context. However, it does not explicitly discuss when to use this versus alternatives, though siblings are not direct alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete_environmentA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description significantly expands on the annotations by explaining what happens: the environment is no longer usable, its plan slot is freed, and its snapshots and history survive. This adds valuable behavioral nuance beyond the destructiveHint=true and idempotentHint=true annotations, and it does not contradict them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the main action, and every clause adds relevant meaning. There is no repetition, filler, or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema, the description covers the action, consequences, and non-destructive nature of data. However, it lacks guidance on how to specify the environment parameter and does not explicitly contrast it with related tools, leaving small but real gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'environment' has no schema description and the description gives no detail about what form it should take (name, ID, path, etc.). Since schema description coverage is 0%, the description was expected to compensate, but it only says 'an environment' without specifying how to identify it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Archive' and clearly identifies the resource ('an environment') and the key consequence: freeing the plan slot it holds. It also distinguishes itself from the sibling delete/create tools by explicitly clarifying that nothing is deleted, which prevents confusion with the tool's name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool: when you want an environment to stop counting as live and free its plan slot, while preserving snapshots and history. It does not explicitly name alternatives or state when not to use it, but the context is clear enough for a simple one-parameter archiving operation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_environmentsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already express readOnly, openWorld, idempotent, and non-destructive behavior. The description adds behavioral context by scoping to the current account and revealing that fork counts are included in the response. This goes beyond the annotations without contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that is front-loaded with the verb 'List' and immediately identifies the resource and output. Every word adds value; there is no unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has no parameters, no output schema, and strong annotations, this description fully covers the core invocation and expected result. An agent has enough information to correctly select and call this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool takes zero parameters, so the baseline of 4 is appropriate. The description does not need to define parameter meanings because there are none to clarify.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'List' and states the exact resource ('this account's environments') plus a meaningful differentiator ('with how many forks each has'). This clearly distinguishes it from get_environment, which focuses on a singular environment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool—when you need an account-level summary of environments and their fork counts—but it does not explicitly say when to prefer this over get_environment or list other alternatives. It provides clear context but no exclusionary guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_environmentA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the read-only, idempotent, and non-destructive annotations, the description adds valuable behavioral context: running processes do not survive a cold boot while files do. This is exactly the kind of contextual insight that the annotations alone do not provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: two sentences front-load the core output details and then add the essential cold-boot behavior. Every sentence earns its place without unnecessary restatement.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter, read-only inspection tool, the description is largely sufficient: it names the returned concepts and a key behavioral nuance, and the annotations cover safety and idempotency. It does not describe a concrete output format, but no output schema is present.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema exposes one undocumented string parameter, 'environment', and schema description coverage is 0%. The description clarifies that the tool operates on an environment's resumption state, but it does not explain how to obtain or format the environment identifier, e.g. whether it is an ID or name from list_environments.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies what the tool returns: an environment's resumption state, including its head snapshot, log progress, and fork tree. This distinguishes it from siblings like list_environments, create_environment, and delete_environment, which perform different lifecycle operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit alternatives are listed, but the description provides clear operational context: every start is a cold boot from the head snapshot, so this tool helps you understand what state an environment will resume with. The absence of explicit when-not-to-use guidance prevents a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_credit_balanceA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and non-destructive behavior. The description adds valuable context beyond annotations: the definition of a credit (one minute per standard environment) and that paused environments don't consume credits. This enriches understanding of the resource being measured without contradicting the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose and followed by a concise clarification of credit units. Every sentence adds essential information with no waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only, zero-parameter tool with rich annotations and no output schema, the description fully covers what the tool does, the resource it references, and the consumption model. This is sufficient for an agent to decide when to call it and what to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description doesn't need to elaborate on parameters but correctly implies that no inputs are required, making it clear the action is a simple balance query.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Show this account's remaining compute credits.' It uses a specific verb (show) and resource (compute credits), and the additional detail about credit usage distinguishes it from sibling tools like create_environment and run_command.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool by explaining how credits are consumed, which helps an agent understand that checking balance is relevant when considering environment execution. It doesn't explicitly mention alternatives or exclusions, but since no sibling tool covers credits, the usage context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.