Server Quality Checklist

Tool Scores

• pingC
  2.1/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. 'Echo' implies a simple read-only operation that returns input unchanged, but it does not specify whether this involves network latency, error handling, or any side effects. The description fails to add context beyond the basic implication, leaving behavioral traits like performance or limitations undocumented.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single word, 'Echo', which is extremely concise and front-loaded with no wasted text. While it may be under-specified, it efficiently communicates the core idea without unnecessary elaboration, earning full marks for brevity and structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional parameter, no output schema, no annotations), the description is incomplete. It does not explain what 'Echo' entails operationally, such as whether it returns the input verbatim or processes it. For even a basic tool, more context on behavior and purpose would improve agent understanding, making the current description inadequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'prompt' documented as 'Message to echo'. The description 'Echo' aligns with this but adds no further meaning beyond what the schema provides, such as examples or constraints. With high schema coverage, the baseline score of 3 is appropriate, as the description does not compensate but also does not detract from parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Echo' is a tautology that essentially restates the tool name 'ping' without adding meaningful clarification. While both terms imply returning what was sent, the description fails to specify what resource or action is involved (e.g., echoing a message parameter). It does not distinguish this tool from potential siblings like 'ask-codex' or 'fetch-chunk' that might also involve response generation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. There is no mention of context, prerequisites, or comparisons to sibling tools such as 'ask-codex' for queries or 'timeout-test' for testing. Without any usage instructions, the agent lacks direction on appropriate scenarios for invoking this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• HelpC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure but provides almost none. 'receive help information' doesn't indicate whether this is a read-only operation, what format the help comes in (text, structured data, links), whether it requires authentication, or any rate limits. The description fails to compensate for the absence of annotations with meaningful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise at just three words, but this brevity comes at the cost of being under-specified rather than efficiently informative. While there's no wasted text, the description fails to provide the minimal necessary information about what the tool actually does. It's more sparse than appropriately concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complete absence of annotations and output schema, the description is inadequate for understanding this tool's functionality. 'receive help information' doesn't explain what help is provided, how it's structured, what topics it covers, or how it differs from other assistance tools in the sibling set. For a tool that presumably provides important guidance to users, this description leaves too many questions unanswered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters with 100% schema description coverage, so the baseline score for this dimension is 4. The description doesn't need to explain parameters since none exist, and it doesn't incorrectly suggest parameters where none are defined. This is appropriate for a parameterless tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'receive help information' is a tautology that essentially restates the tool name 'Help' without providing meaningful specificity. It doesn't explain what kind of help information is provided, in what format, or what resources it covers. While it's clear this is a help tool, it lacks the verb+resource specificity that would distinguish it from other help mechanisms.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance about when to use this tool versus alternatives. With sibling tools like 'ask-codex', 'brainstorm', and 'fetch-chunk' that might also provide assistance or information, there's no indication whether this is a general help system, documentation lookup, or something else. The agent receives no explicit or implied context about appropriate usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ask-codexC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'safety controls' and 'changeMode' but doesn't explain what safety controls exist, what risks are involved, what permissions are needed, or what the tool actually does behaviorally. The description is too vague about execution behavior, error handling, or output format to be helpful for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently mentions key capabilities. It's appropriately sized for a complex tool, though it could be more front-loaded with the core purpose. There's no wasted verbiage, but the structure is basic without clear separation of concerns.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with 23 parameters, no annotations, and no output schema, the description is inadequate. It doesn't explain what the tool returns, what kind of execution occurs, what safety considerations exist, or how it differs from sibling tools. The description fails to compensate for the lack of structured metadata about this significant CLI execution tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 23 parameters thoroughly. The description adds minimal value beyond what's in the schema - it mentions 'file analysis (@syntax), model selection, and safety controls' which correspond to some parameters, but doesn't provide additional semantic context or usage patterns. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Execute Codex CLI with file analysis (@syntax), model selection, and safety controls. Supports changeMode.' This provides a general purpose (executing Codex CLI) with some features mentioned, but it's vague about what Codex CLI actually does and doesn't distinguish it from sibling tools like 'batch-codex' or 'brainstorm'. The description mentions capabilities but lacks specificity about the core function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when this tool is appropriate compared to 'batch-codex' (presumably for batch processing) or 'brainstorm' (presumably for ideation). There's no context about prerequisites, typical use cases, or exclusions for this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• brainstormC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'feasibility analysis' but lacks critical details: it doesn't specify whether this is a read-only or mutating operation, what permissions or authentication might be required, potential rate limits, or output format. For a tool with 17 parameters and no annotations, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose without unnecessary words. Every phrase ('creative ideas', 'structured frameworks', 'domain context', 'feasibility analysis') contributes meaningfully, making it appropriately sized and well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (17 parameters, no annotations, no output schema), the description is incomplete. It doesn't address behavioral aspects like safety, permissions, or output format, and while schema coverage is high, the description itself lacks depth to guide an agent in using such a multifaceted tool effectively. This is inadequate for a tool of this scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, meaning all parameters are documented in the schema itself. The description adds minimal value beyond the schema by hinting at 'structured frameworks' (related to 'methodology') and 'domain context' (related to 'domain'), but doesn't provide additional syntax, format, or usage details for parameters. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Generate creative ideas using structured frameworks with domain context and feasibility analysis.' It specifies the verb ('generate'), resource ('creative ideas'), and key aspects ('structured frameworks', 'domain context', 'feasibility analysis'). However, it doesn't explicitly differentiate from sibling tools like 'ask-codex' or 'batch-codex', which might also generate content, so it doesn't reach the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'ask-codex' or 'batch-codex', nor does it specify contexts or exclusions for usage. The agent must infer usage based on the purpose alone, which is insufficient for effective tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• timeout-testC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool 'runs for a specified duration' to test timeout prevention, implying it simulates a time-consuming operation. However, it lacks details on potential side effects (e.g., resource consumption), error handling, or what constitutes a successful test. For a tool with no annotations, this is a significant gap in behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence: 'Test timeout prevention by running for a specified duration.' It is front-loaded with the core purpose, has zero wasted words, and is appropriately sized for a simple tool with one parameter. Every part of the sentence earns its place by conveying essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (simple with one parameter) and the absence of annotations and output schema, the description is incomplete. It explains what the tool does but lacks context on how it integrates with sibling tools, what the expected outcome is (e.g., success/failure indicators), or any behavioral nuances. For a testing tool, more guidance on interpretation would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'duration' parameter fully documented in the schema (type, minimum, description). The description adds no additional meaning beyond the schema, as it only mentions 'a specified duration' without elaborating on units or constraints. Given the high schema coverage, the baseline score of 3 is appropriate, as the description doesn't compensate but also doesn't detract.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Test timeout prevention by running for a specified duration.' It specifies the verb ('Test timeout prevention') and resource/action ('running for a specified duration'), making it clear what the tool does. However, it doesn't explicitly distinguish itself from sibling tools like 'ping' or 'version', which might also test system functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention any prerequisites, context, or exclusions, and there's no comparison to sibling tools like 'ping' (which might test connectivity) or 'ask-codex' (which might involve processing). This leaves the agent with minimal direction on appropriate usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• batch-codexB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure but offers minimal details. It mentions 'batch processing' and use cases but doesn't describe critical behaviors like execution flow, error handling, output format, or resource implications. For a complex tool with 11 parameters and no annotations, this is inadequate, though not contradictory.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two sentences that efficiently convey purpose and ideal use cases. Every word earns its place without redundancy, and it's front-loaded with the core function. No unnecessary elaboration or waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (11 parameters, batch processing, no annotations, no output schema), the description is insufficient. It lacks details on execution behavior, result format, error handling, and integration with sibling tools. While concise, it doesn't provide enough context for an agent to fully understand how to invoke and interpret this tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all 11 parameters. The description adds no specific parameter details beyond implying tasks involve 'atomic' operations and targets use '@ syntax.' This meets the baseline of 3 since the schema does the heavy lifting, but the description doesn't enhance understanding of parameter interactions or semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Delegate multiple atomic tasks to Codex for batch processing' with specific use cases like 'repetitive operations, mass refactoring, and automated code transformations.' It distinguishes from sibling tools like 'ask-codex' by emphasizing batch processing rather than single interactions. However, it doesn't explicitly contrast with all siblings, preventing a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides implied usage context with 'Ideal for repetitive operations, mass refactoring, and automated code transformations,' which suggests when to use this tool. However, it lacks explicit guidance on when NOT to use it or clear alternatives among siblings like 'ask-codex' for single tasks. No prerequisites or exclusions are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• versionB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the tool's behavior as a display operation, implying it's read-only and non-destructive, but doesn't add context like output format, rate limits, or authentication needs. This is adequate for a simple tool but lacks depth for richer behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose without any waste. Every word earns its place by directly conveying what the tool does, making it highly concise and well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is complete enough to convey basic functionality. However, it lacks details on output format or system information specifics, which could be helpful for an agent. It's minimally viable but has clear gaps in richer context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100%, so no parameter documentation is needed. The description doesn't add parameter semantics, which is appropriate here. A baseline of 4 is applied as it meets expectations for a parameterless tool without unnecessary details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Display') and resource ('version and system information'). It distinguishes this from siblings like 'ping' or 'Help' by focusing on system metadata rather than connectivity or assistance. However, it doesn't explicitly differentiate from all siblings (e.g., 'timeout-test' might also relate to system behavior).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description doesn't mention prerequisites, appropriate contexts, or comparisons to sibling tools like 'ping' for basic connectivity checks or 'Help' for documentation. Usage is implied only by the tool's name and purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch-chunkA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses that this is a retrieval operation for cached data, implying read-only behavior, but doesn't mention potential limitations like cache expiration, rate limits, or error conditions. The description adds some context about the tool's role in a multi-step process but lacks comprehensive behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with zero waste - the first states the purpose, the second provides usage guidance. Every word earns its place, and the information is front-loaded with the core function stated immediately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 required parameters, no output schema, no annotations), the description is reasonably complete. It explains the tool's purpose, when to use it, and its relationship to changeMode responses. However, without annotations or output schema, it could benefit from more detail about what the retrieved chunks contain or potential limitations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents both parameters. The description adds minimal value beyond what the schema provides - it mentions 'cache key provided in the initial changeMode response' which slightly clarifies the cacheKey parameter's origin, but doesn't add significant semantic context beyond the schema's descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Retrieves cached chunks') and resource ('from a changeMode response'), distinguishing it from sibling tools like ask-codex or brainstorm. It precisely defines the tool's function as fetching subsequent data chunks after an initial partial response.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use this tool ('Use this to get subsequent chunks after receiving a partial changeMode response'), providing clear context and timing guidance. It distinguishes this from initial retrieval tools, though it doesn't name specific alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.