Server Quality Checklist

Tool Scores

• run_allA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate destructive/open-world behavior, and the description supplements this with concrete details: 'each can read/edit files in cwd' and 'one confirmation covers the whole batch.' It also mentions performance and quota impacts. This adds value beyond the provided annotations without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the primary action, followed by a purpose clause and a caveat-rich second sentence. Every phrase earns its place with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core purpose, return format, file access, confirmation behavior, and performance implications. Given the rich schema and annotations, this is a solid overview. It does not mention every detail (e.g., model fallback rules or timeout mechanics), but those are fully documented in the schema, so the description is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and each parameter already has a detailed description (e.g., cwd is 'not a sandbox', model formatting rules, timeout semantics). The tool description only references cwd behaviorally but doesn't add new meaning beyond the schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Fan the SAME prompt out to every enabled agent concurrently and return each agent's answer, labelled per agent.' It uses a specific verb ('fan out') and resource ('every enabled agent'), and distinguishes this from sibling tools like codex/claude, which target a single agent.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives context on when to use the tool: 'for comparing agents or cross-checking a result.' It also warns about costs: 'can be slow or use several agents' quotas.' However, it does not explicitly name an alternative tool for single-agent use, so it falls just short of a perfect 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• claudeA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnlyHint=false, openWorldHint=true, destructiveHint=true), the description discloses the non-interactive execution, file read/edit capabilities, potential time and quota usage, and a detailed error-handling scheme (classified errors for common failures, clipped tail for others). This significantly enriches the agent's understanding of side effects and limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences, each carrying distinct information: what the tool is, how it executes, how failures are handled, and a prerequisite. There is no redundancy, and the most important facts are front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, execution, error behavior, and prerequisites, which is quite complete for a complex tool with no output schema. It briefly explains the return value ('returns its output' and failure modes), though a more detailed description of the successful output format would make it fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description mentions 'cwd' contextually but does not add semantics beyond what the schema already provides; the schema itself fully describes parameters like timeoutMs, model, and prompt.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Runs the `claude` CLI non-interactively in `cwd`' for 'complex implementation, refactoring, and code review'. This specifies both the verb and resource, and identifies it as 'Anthropic's coding agent', distinguishing it from sibling tools like codex or run_all.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context for when to use: for complex coding tasks requiring implementation, refactoring, or code review. It also gives a specific prerequisite ('Check availability with list_agents first') and warns about time and model quota costs, which helps the agent decide. However, it does not explicitly contrast with alternatives like codex, so it lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• review_changeA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes well beyond the annotations by detailing the exact sequence of operations (runner edits → diff captured → reviewer judges → verdict returned). It also discloses specific edge-case behaviors: untracked files are reviewed by contents (bounded per file) and the diff may include pre-existing dirty changes. This adds substantial value beyond the readOnlyHint/destructiveHint flags.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: three sentences that explain the core workflow, a prerequisite, and two important edge-case caveats. It is front-loaded with the primary action and avoids redundancy, earning its place with no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multi-agent, 8 params, no output schema), the description covers the key aspects: operational flow, return contents (verdict, runner output, diff, review), and environmental requirements. However, it could more explicitly state that this is a persistent-mutation operation and when to avoid it, though annotations handle the safety flag partially.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 8 parameters have descriptions in the schema, so the baseline is 3. The tool description reinforces the role of `runner` and `reviewer` (one edits, one judges) but does not add meaning beyond what the schema already states. No additional parameter context is provided beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the tool's purpose: run the `runner` agent to edit files, capture the `git diff`, then have the `reviewer` agent judge it and return a verdict with outputs. This distinguishes it from sibling tools like `codex` or `run_all` by describing its unique multi-step review workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool—when you need an agent to make a change and have another agent review it. It also gives a necessary prerequisite ('Requires a git worktree') and warns about pre-existing changes in a dirty worktree, but it does not explicitly name alternatives or spell out when not to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codexA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already flag destructive and open-world behavior, but the description adds rich context: non-interactive execution, ability to read/edit any accessible files (not sandboxed), time/model quota usage, classified failure modes, and clipped stderr/stdout on non-zero exits. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three dense sentences: purpose, execution behavior, failure handling, and availability check. Information is front-loaded with tool identity, and every sentence earns its place without fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description covers success output ('returns its output'), failure classification, and operational caveats. Combined with strong schema and annotations, the tool is fully specified for an agent to use safely.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 5 parameters have detailed schema descriptions (100% coverage) covering cwd resolution, model override validation, and timeout semantics. The description itself adds no extra parameter detail, so the baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Runs the codex CLI non-interactively in cwd' and explains it 'autonomously reads and edits files to implement, refactor, and fix code.' This clearly distinguishes it from siblings like claude and run_all by naming the specific agent and its function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs to 'Check availability with list_agents first' and clearly implies when to use: when you need OpenAI Codex to autonomously work on code. It does not explicitly compare to claude or run_all, but the use case is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pingA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and openWorldHint=false. The description adds the concrete response ('returns "pong"') and reinforces the side-effect-free nature. It doesn't repeat or contradict annotations, and provides sufficient behavioral context for a zero-parameter operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence. It states the purpose, the return value, and the safety profile without any redundant or extraneous text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a trivial ping tool with no parameters, no output schema, and annotations covering read-only behavior, the description is complete. An agent has everything needed to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema is trivially complete (100% coverage). The description adds no parameter details because none are needed. Baseline 4 is appropriate for a no-parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: 'Liveness check for agent-mcp-hub' and what it returns ('pong'). This distinguishes it from sibling tools like list_agents or run_all, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The context is clear: it's a liveness check, so an agent knows to use it to verify the server is reachable. It doesn't explicitly mention when not to use it or name alternatives, but given the tool's simplicity and distinct purpose among siblings, the usage is well implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_agentsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite the readOnlyHint annotation, the description adds valuable nuance: it explains the exact meaning of installed vs usable, and the concrete pitfall where codex --version exits 0 despite an unwritable home. This goes beyond the annotation's bare safety flag and clarifies what 'can actually run' means.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every sentence serves a purpose: purpose, field meanings, a critical caveat, and usage direction. It is front-loaded with a clear statement of what the tool does and remains compact despite covering nuanced semantics.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description fully specifies return fields (installed, usable, reason, available) and their semantics, including the installed-but-unusable distinction. The practical guidance to prefer usable completes the picture for choosing an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool accepts zero parameters, so the schema already fully documents the input. The description focuses instead on return fields, which is appropriate; no parameter clarification is needed beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb-resource pair ('List the wrapped coding-agent CLIs') and adds scope ('whether each can actually run'), clearly distinguishing it from sibling agent-execution tools like codex/claude. The elaboration about installed vs usable further pins down its function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs 'call this first to choose an agent before delegating,' establishing when to invoke it relative to sibling tools. It also advises 'prefer usable' over installed, giving evaluation guidance for interpretation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.