Server Quality Checklist

Tool Scores

• analyze_securityC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions analysis components but doesn't disclose behavioral traits such as execution time, resource intensity, permission requirements, whether it modifies data, or output format. For a security analysis tool with no annotation coverage, this leaves significant gaps in understanding how it behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the main purpose and lists key analysis components. It avoids redundancy and wastes no words, though it could be slightly more structured for readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of security analysis, no annotations, and no output schema, the description is incomplete. It lacks details on what the analysis entails operationally, how results are returned, error handling, or integration with sibling tools. This makes it inadequate for an agent to fully understand the tool's context and usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters (owner, repo, branch) with clear descriptions. The description adds no additional meaning about parameters beyond implying repository context. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs 'comprehensive security analysis on a repository' and lists specific components (secret detection, vulnerability assessment, risk scoring, compliance status). It distinguishes from siblings like 'get_secret_types' (focused only on secret types) and 'scan_code' (likely narrower scope), though not explicitly. However, it doesn't fully differentiate from 'scan_repository' which might overlap.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like 'scan_repository', 'scan_code', or 'validate_secret' is provided. The description implies a comprehensive analysis but doesn't specify prerequisites, timing, or exclusions (e.g., when a simpler scan might suffice).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_secretB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool validates secret status but does not explain how it performs this validation (e.g., via API calls, internal checks), potential side effects (e.g., logging, rate limits), or error handling. For a tool that likely interacts with external systems, this lack of detail is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that directly states the tool's purpose without unnecessary words. It is front-loaded and efficient, making it easy to understand at a glance. Every part of the sentence earns its place by conveying essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (validating secrets, likely involving external checks), the lack of annotations and output schema means the description should provide more context. It adequately states what the tool does but fails to cover behavioral aspects like validation methods or result formats. For a tool with no structured safety or output information, this is a minimal but incomplete description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, clearly documenting both parameters ('secretType' and 'value') with examples. The description does not add any additional meaning beyond the schema, such as format details or validation rules. According to the rules, with high schema coverage, the baseline score is 3, as the schema adequately handles parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Validate if a detected secret is still active or has been revoked.' It specifies the verb 'validate' and the resource 'detected secret,' with the outcome being a status check. However, it does not explicitly differentiate this from sibling tools like 'analyze_security' or 'scan_code,' which might also involve secret-related operations, leaving some ambiguity about its unique role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention prerequisites, such as needing a previously detected secret, or compare it to sibling tools like 'get_secret_types' for listing types or 'scan_repository' for initial detection. Without such context, users must infer usage based on the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_secret_typesB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states this is a read operation ('Get list'), implying it's non-destructive, but doesn't disclose other behavioral traits like rate limits, authentication needs, or response format. The mention of '35+ types' hints at a static list, but more context would help.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words. It front-loads the key action and resource, and the additional context ('35+ types', 'that can be detected') earns its place by clarifying scope. This is appropriately sized for a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is minimally adequate. It explains what the tool returns (a list of secret types) but lacks details on format, ordering, or how it integrates with siblings. For a read-only tool, this is passable but could be more complete with behavioral context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description doesn't add parameter details, which is appropriate here. A baseline of 4 is given as it compensates adequately for the lack of parameters by focusing on the tool's purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get list') and resource ('all supported secret types'), specifying what the tool does. It adds useful context about quantity ('35+ types') and purpose ('that can be detected'). However, it doesn't explicitly differentiate from sibling tools like 'validate_secret' or 'scan_code', which might also involve secret types.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, context (e.g., before scanning or validation), or exclusions. With siblings like 'analyze_security' and 'scan_repository', users are left to infer usage based on the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_codeA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses behavioral traits such as the 10MB size limit and 'enhanced validation,' but lacks details on rate limits, authentication needs, output format, or error handling. This is a moderate disclosure given the absence of annotations, but more context would improve transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, consisting of two concise sentences that directly state the tool's purpose and key constraints. Every sentence earns its place by providing essential information without redundancy, making it efficient and easy to understand.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (scanning for secrets with size limits), no annotations, and no output schema, the description is moderately complete. It covers the purpose and size limit but lacks details on output format, error cases, or integration with sibling tools. This is adequate for basic use but has clear gaps for full contextual understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('code' and 'filePath') with descriptions. The description adds minimal value beyond the schema by implying the 'code' parameter handles up to 10MB, but does not provide additional syntax or format details. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('scan') and resource ('code snippet or file content'), specifying what it scans for ('secrets and sensitive information'). It distinguishes from sibling tools like 'scan_repository' by focusing on code content rather than repositories, and from 'validate_secret' by scanning for secrets rather than validating them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by mentioning 'supports up to 10MB of code content with enhanced validation,' suggesting it's for scanning code up to that size. However, it does not explicitly state when to use this tool versus alternatives like 'scan_repository' for repositories or 'analyze_security' for broader analysis, nor does it provide exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_repositoryA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively adds context beyond basic functionality by mentioning 'Enterprise-grade with rate limiting, retry logic, and comprehensive error handling,' which informs the agent about performance traits and reliability. However, it doesn't specify authentication requirements or potential side effects, leaving some gaps in behavioral understanding.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences: the first states the core purpose with specific details, and the second adds behavioral context. Every sentence earns its place by providing essential information without redundancy, making it front-loaded and appropriately sized for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (security scanning with behavioral traits) and the absence of annotations and output schema, the description is moderately complete. It covers purpose and some behavioral aspects but lacks details on output format, error specifics, or integration with sibling tools. This is adequate for basic use but leaves room for improvement in guiding the agent fully.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all four parameters (owner, repo, branch, path) with clear descriptions. The description adds no additional parameter semantics beyond what the schema provides, such as examples or constraints, but it doesn't need to compensate for gaps. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Scan a GitHub repository') and resource ('for secrets and sensitive information'), distinguishing it from sibling tools like 'analyze_security' or 'scan_code' by focusing specifically on secret detection. It provides concrete details about what it detects ('35+ types of secrets including API keys, passwords, tokens, and credentials'), making the purpose highly specific and differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'analyze_security' or 'scan_code', nor does it mention any prerequisites or exclusions. While it implies usage for security scanning, it lacks explicit context for tool selection, leaving the agent to infer based on tool names alone without clear differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.