Server Quality Checklist

Tool Scores

• codex_auth_statusA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states it 'never returns a token, API key, cookie, or any other secret,' indicating a read-only, safe operation. It also notes the interactive login flow cannot be triggered, and explains the two auth modes. It adds value beyond the basic purpose statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise but includes necessary details. It front-loads the core purpose, then provides usage context, security guarantees, auth mode details, and troubleshooting steps. Each sentence adds value without redundancy, though it could be slightly trimmed without losing meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no params, no annotations, no output schema), the description is remarkably complete. It explains what the response will report (active mode, configured mode, agreement), what to do on authentication failure, and the constraints (no secrets, interactive login unavailable). It adequately prepares the agent for safe and correct use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the input schema is an empty object. The description does not need to explain parameters, so it earns the baseline 4. It does not introduce any param-specific details, but none are required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Report whether Codex is authenticated and which auth mode is in use.' It uses a specific verb (report) and resource (authentication status), and the mention of auth modes distinguishes it from sibling tools like codex_qualify and codex_capabilities, which likely serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises when to call the tool: 'Call this before a review if you want to fail fast with a clear message.' It also explains what to do if authentication fails (run codex-mcp login). However, it does not mention alternatives or situations where the tool should not be used, so it lacks full exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codex_qualifyA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explicitly states the tool is read-only ('cannot edit files, commit, push, modify issues, or write to any database or external system'), describes what it returns ('review delta'), and explains the review process (inspects repository, derives expected coverage, compares). This fully discloses behavioral traits without needing annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear paragraphs: purpose, usage instructions, process explanation, and side-effect transparency. It is informative without being verbose, each sentence adds value, and it avoids redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multiple parameters, review types, and candidate structures), the description covers the essential aspects: what it does, how to use it, its read-only nature, and the expected outcome (review delta). It also guides post-review actions ('verify each objection'), making it complete for an agent to decide when and how to invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description clarifies required vs optional parameters: 'Required: reviewType, project.root, and a matching candidate set' and 'Everything else is optional.' This adds meaningful context beyond the schema. However, the schema marks candidate as optional (default {}), while the description implies it is required, a minor inconsistency that slightly reduces clarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: independently qualify candidate test cases and/or bug findings. It uses specific verbs (qualify, inspect, compare) and identifies the resource (candidate test cases/bugs) and distinguishes itself from sibling tools (auth/capabilities) by focusing on review.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit usage timing: 'before you write your final artifact' and 'do not call this after the report is published.' It also instructs to send the candidate in memory rather than writing to a file first, and clarifies that it is a second opinion, giving clear when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codex_capabilitiesA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Given no annotations, the description carries full behavioral burden. It discloses that it runs no review, does not modify anything, may take a moment to probe connectors, and supports a configuration-only mode via the probeConnectors parameter.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact yet information-dense. It opens with a clear statement of purpose, then covers usage, side effects, and a practical parameter tip—all in two short paragraphs with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a diagnostic tool with no output schema, the description adequately explains what will be reported (reachable evidence, forbidden actions, withheld tools) and the optional parameter's effect. It is complete for an agent to decide when and how to invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although the schema already describes the parameter with 100% coverage, the description adds meaningful nuance: the time trade-off of probing and the option to disable it for a faster configuration-only answer. This goes beyond the schema's static description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reports what evidence it can reach and what it is forbidden to do, using a specific verb ('Report') and resource ('this codex-mcp instance'). It distinguishes itself from sibling tools like codex_qualify and codex_auth_status by focusing on capabilities and policy restrictions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly tells the agent when to use it: 'before you rely on it' to check availability of the requirement system or database and which tools were withheld. Also states it is diagnostic-only and changes nothing, setting expectations for use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.