Server Quality Checklist

Tool Scores

• apply_winnerA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description must carry the transparency burden. It does well by explaining the re-verification on a fresh clean fork, the receipt concept, and the return value. It could be more explicit about failure behavior and exact side effects on the local tree, but core behavioral traits are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The purpose, verification step, and return shape are all covered in a tight, front-loaded structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no annotations and 0% parameter coverage, the description needs to do more heavy lifting. It explains the main flow and return value, but leaves session_id semantics, test_cmd usage, failure behavior, and sibling-tool routing unaddressed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so parameter meaning must come from the description. It implies fork_id via 'one fork's diff,' repo_path via 'local tree,' and test_cmd via 're-verifies,' but session_id is never explained and no parameter is explicitly mapped to its schema property.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Land one fork's diff on the local tree,' a concrete verb-object statement that clearly names the resource and target. It does not explicitly contrast with siblings like try_all or try_many, so it falls just short of full differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: use this tool when a specific fork's diff should be landed on the local tree. It does not state when not to use it or mention alternatives like try_all/try_many, but the intended usage is apparent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sessionsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden of behavioral disclosure. It states that the tool lists sessions and their forks, implying a read-only operation, but it does not explicitly declare the lack of side effects, authentication requirements, or any potential pagination. For a zero-parameter list tool, this is acceptable but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no filler. The action verb is front-loaded, and the content is directly informative. It achieves maximum clarity with minimal words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no parameters and a defined output schema, the description provides enough information for an agent to call it correctly. It names the core output fields, which is sufficient. Adding explicit read-only language or ordering/filtering notes would be a minor improvement but not necessary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description needs to add no parameter-specific semantics. The baseline for 0 parameters is 4, and the description adequately covers the tool's purpose without needing any parameter explanation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('List'), a clear resource ('superpose sessions and their forks'), and the exact fields returned ('status, timing, cost'). This makes the tool's purpose unambiguous and differentiates it from the sibling tools, which likely focus on executing actions rather than reviewing existing sessions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'for review' provides clear context that this tool is meant for inspecting or auditing sessions, which implies it should be used when the agent needs an overview rather than taking an action. However, it does not explicitly name alternatives or state when not to use it, so it stops short of a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• try_manyA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden and handles it well: it discloses parallel execution, live concurrent Monitor display, separate branch trees, and that every session's result is returned. It does not discuss side effects, permissions, or cleanup, but the branch-tree model implies isolation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three dense sentences with no fluff: purpose and usage are front-loaded, parameter shape comes second, and return/monitor behavior closes it out. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the parallel execution model, the task object shape, the monitor URL, and the return behavior; the presence of an output schema also covers detailed return structure. It relies on the sibling definition of try_all for full field semantics and omits explicit side-effect or prerequisite warnings, so it is not flawless.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description's breakdown of `tasks` as a list of `{task, approaches, test_cmd?, benchmark_cmd?}` objects is essential and useful. `repo_path` is only indirectly described via 'same repo', which is adequate given the parameter name but less explicit.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource: 'Run several DIFFERENT tasks on the same repo AT ONCE—each becomes its own parallel Superpose session.' This makes the tool's purpose unmistakable and differentiates it from siblings like try_all, which is referenced as the source of the task-field format rather than as the same behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use the tool: 'Use this when you have multiple independent changes queued for one codebase and want them worked in parallel from a single call.' It does not explicitly list counter-indications or name alternative tools, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• try_allA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden, and it delivers: it discloses fork isolation, execution in parallel, what each fork includes, the outputs (diff, test result, wall-clock, cost), the benchmark ordering after tests pass, the expected metric format, and winner selection. This is rich behavioral detail beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured in three short paragraphs, front-loading the core purpose and outputs. Every sentence adds meaningful guidance: the example clarifies strategy diversity, the benchmark paragraph adds precise operational detail, and the final pointer to apply_winner closes the workflow.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core workflow, constraints, benchmark semantics, and follow-up action; an output schema exists so return-value details need not be spelled out. The only notable gap is that it does not state what happens when no 'benchmark_cmd' is supplied, i.e., whether there is still a winner or how the agent should decide.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It clearly explains 'approaches' (2-8, short label + instruction) and 'benchmark_cmd' (runs after tests, prints SUPERPOSE_METRIC, lower is better). It does not explicitly map every parameter name like 'repo_path' or 'test_cmd' to its schema property, though their roles are implied strongly by context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description leads with a specific verb and resource: 'Run several genuinely different strategies... each in its own forked machine' and explicitly lists what is returned. It also separates itself from 'apply_winner' by framing that tool as the follow-up step, making the purpose and boundary clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives an explicit when-to-use rule: 'Call this instead of picking one approach yourself whenever a task has more than one plausible strategy,' with a concrete example. It also specifies the 2-8 approach limit, how each approach should be phrased, and when to use the winner with 'apply_winner.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.