Server Quality Checklist

Tool Scores

• install_base_serversC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool performs an installation, implying a write/mutation operation, but fails to describe critical behaviors such as whether it overwrites existing configurations, requires specific permissions, or has side effects like restarting services. This leaves significant gaps in understanding the tool's impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, straightforward sentence that efficiently conveys the core action without unnecessary words. It is appropriately sized for a simple tool, though it could be more front-loaded with additional context to improve clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with no annotations and no output schema, the description is incomplete. It lacks details on what 'base MCP servers' entail, the outcome of the installation, error conditions, or how it interacts with the configuration file. This leaves the agent with insufficient information to use the tool effectively in complex scenarios.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, clearly documenting the single parameter 'configPath'. The description does not add any meaning beyond what the schema provides, as it mentions no parameters. Given the high schema coverage, the baseline score of 3 is appropriate, as the schema adequately handles parameter semantics without extra help from the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the action ('Install') and target ('base MCP servers'), but it's vague about what 'base MCP servers' specifically are and doesn't distinguish this from sibling tools like analyze_code or collect_code. It provides a basic purpose but lacks specificity and differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives or in what context it should be applied. The description does not mention prerequisites, timing, or exclusions, leaving the agent with no usage instructions beyond the basic action.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_codeC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses that analysis 'may take a few minutes' (timing behavior) and requires an API key (authentication need), which are useful behavioral traits. However, it lacks details on rate limits, error handling, or what happens during analysis (e.g., data sent to OpenAI). No contradiction with annotations exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three short sentences, front-loading the main action ('Analyze code using OpenAI API'). However, the phrase 'So, wait please' is slightly informal and could be more structured, though it efficiently conveys timing without unnecessary details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and a simple input schema, the description provides basic context (purpose, timing, auth) but is incomplete. It doesn't explain what the analysis returns (e.g., insights, reports) or potential limitations, leaving gaps for a tool that interacts with an external API and involves waiting.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'codePath' well-documented in the schema as an absolute path. The description adds no additional meaning beyond this, such as file format constraints or analysis scope based on path. Baseline 3 is appropriate as the schema handles parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Analyze code using OpenAI API' which provides a verb ('analyze') and resource ('code'), but it's vague about what analysis entails (e.g., security, performance, style). It doesn't distinguish from siblings like 'collect_code' or 'create_github_issues', leaving ambiguity in purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions 'requires your API key' and 'wait please', implying prerequisites and timing, but offers no explicit guidance on when to use this tool versus alternatives like 'collect_code' for gathering code or 'create_github_issues' for issue tracking. No exclusions or clear context for tool selection are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• collect_codeC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the action ('collect') and output format ('markdown file'), but fails to detail critical behaviors like whether it overwrites existing files, handles errors (e.g., missing directories), requires specific permissions, or includes metadata in the output. This leaves significant gaps for a tool that modifies files.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose without unnecessary details. It uses clear language ('collect all code', 'single markdown file') and avoids redundancy, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (file system operations, output generation) and lack of annotations or output schema, the description is incomplete. It doesn't cover behavioral aspects like error handling, file overwriting, or output structure, which are crucial for safe and effective use. This inadequacy is notable for a tool that creates files.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal parameter semantics beyond the schema, which has 67% coverage. It implies 'input' is for source paths and 'outputPath' for the markdown file, but doesn't explain the dual nature of 'input' (directory vs. list) or how 'ignorePatterns' functions in practice. With moderate schema coverage, the baseline is 3, as the description doesn't fully compensate for the gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('collect all code') and the output ('into a single markdown file'), specifying both verb and resource. However, it doesn't explicitly differentiate from sibling tools like 'analyze_code' or 'create_github_issues', which might involve code handling but serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'analyze_code' for code analysis or 'create_github_issues' for issue tracking. It lacks context about prerequisites, such as needing access to the directory, or exclusions, like not being suitable for real-time code processing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_github_issuesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the requirement for a GitHub token, which adds some context about authentication needs. However, it fails to describe key behavioral traits such as whether this is a write operation (implied by 'create' but not explicit), potential side effects, error handling, rate limits, or what the output looks like. For a mutation tool with zero annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two short sentences that are front-loaded with the main purpose. There's no unnecessary verbosity, and each sentence serves a purpose: the first states the action, and the second adds a critical requirement. However, it could be slightly more structured by explicitly separating purpose from prerequisites.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity as a write operation with no annotations and no output schema, the description is incomplete. It lacks details on behavioral aspects like mutation effects, error cases, and output format. While it mentions a token requirement, it doesn't cover other contextual needs such as permissions or integration with sibling tools, leaving gaps for the agent to infer.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, meaning the input schema already documents all parameters thoroughly. The description adds no additional meaning about parameters beyond what's in the schema, such as clarifying the relationship between 'issueNumbers' and 'analysis results' or providing examples. With high schema coverage, the baseline score of 3 is appropriate as the description doesn't compensate but also doesn't detract.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('create GitHub issues') and the source ('from analysis results'), providing a specific verb+resource combination. However, it doesn't differentiate from sibling tools like 'analyze_code' or 'collect_code' which might also relate to code analysis workflows, leaving room for ambiguity about when to use this versus other tools in the server.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions 'Requires GitHub token' which is a prerequisite but not a usage guideline. It provides no guidance on when to use this tool versus alternatives like 'analyze_code' or 'collect_code', nor does it specify scenarios where this tool is appropriate or inappropriate. Without such context, the agent lacks direction on tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.