Server Quality Checklist

Tool Scores

• scan_for_log_leaksA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions what it detects (patterns like console.log(process.env.SECRET)) but does not disclose whether it is read-only, what happens after detection, or the format of results. This lack of output/behavior detail is a significant gap for a security-related tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two concise sentences. The first states the primary purpose, the second gives concrete examples of patterns caught. No filler or redundant information is present, making it highly efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with no annotations or output schema, the description is reasonably complete in conveying purpose and invocations. However, it lacks any mention of return values or error behavior, which is important for agents to assess results. The examples add value, but the missing output expectations keep it at a middling score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides 100% coverage of the only parameter, projectPath, with a description ('Absolute path to the project root to scan'). The tool description adds no additional detail beyond that, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Scan') and resource ('source files') with a clear focus on console.log / logger calls that leak environment variables or secrets. It distinguishes itself from sibling scan_for_secrets by specifically targeting log statements and runtime leaks, not general secret scanning.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used to catch log leaks before production, but it does not explicitly state when to use this over other siblings like scan_for_secrets or provide exclusions. The context is clear enough, but no alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_for_secretsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the return format (file path, line number, severity, masked preview) and detection categories, which is useful. However, it does not mention whether the scan is read-only, any permissions required, or limitations like ignored directories, leaving some behavioral aspects unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, two sentences long, with the primary purpose front-loaded and additional detail on detected patterns and return values. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description explains what is returned (file path, line number, severity, masked preview). It covers detection scope and examples. Missing some context like hidden files or performance considerations, but overall adequate for a scanner tool of moderate complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, covering both parameters (extensions and projectPath). The description does not add extra meaning beyond the schema, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Scan' with a clear resource 'project directory' and object 'hardcoded secrets, API keys, tokens, and passwords.' It also differentiates from sibling tools like scan_for_log_leaks by focusing on secrets rather than logs or other scans.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (when needing to find secrets in a project directory) but does not explicitly state exclusions or compare with sibling tools. It lacks guidance on when not to use this tool or which alternative to choose for other scanning needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_gitignore_coverageA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility. It discloses the key behavior (flags files that could be accidentally committed) but omits details about read-only nature, return format, or potential false positives. This is adequate for a simple check tool but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two focused sentences: the first defines the tool's purpose with examples, the second explains the flagging behavior. No redundant information or filler words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description only vaguely says 'flags files' without specifying return format or prerequisites. This is minimally sufficient for a single-parameter tool, but additional details about output or limitations would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides a complete description for the only parameter ('Absolute path to the project root'), so the description adds no additional parameter semantics. Baseline 3 applies due to high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool checks whether sensitive files are covered by .gitignore rules, naming specific examples (.env, secrets.json). It distinguishes itself from sibling scan tools by focusing on coverage rather than content scanning.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: to verify sensitive files are protected from accidental commits. It doesn't explicitly name alternatives or exclusions, but the distinction from sibling scan tools is implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_ci_workflowsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full behavioral transparency. It explains the two risk categories (log leak and injection), how the risks arise, and what output fields to expect (file, job, step, pattern_found, risk, recommendation). This goes beyond a typical scan tool description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences deliver substantial detail: file paths, patterns, risk explanations, and return fields. Every clause provides useful context without redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is comprehensive for a scanning tool with no output schema, listing both detection patterns and return fields. It does not address edge cases like repositories without CI files or error handling, but given the sibling context and clarity, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers the single parameter repo_path with a clear description. The tool description itself does not add further parameter details, which is acceptable given 100% schema coverage, but it does not exceed the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Scans' and names exact resources: GitHub Actions, CircleCI, and GitLab CI workflow files. It clearly differentiates from siblings by focusing on CI workflows and specific dangerous interpolation patterns, unlike general secret scanning or log leak tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on what it targets (CI workflow files) and what it detects, but it does not explicitly state when to use this tool over alternatives like scan_for_secrets or scan_for_log_leaks. The specificity implies usage, but no explicit exclusions or alternative comparisons are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_git_historyA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does well: it discloses detection method (diff hunks), output fields, and respect for .gitleaksignore. However, it does not explicitly state read-only behavior or potential performance implications of scanning large histories.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is composed of three efficient, front-loaded sentences. It packs a clear purpose, behavioral details, and a concise list of return fields without any fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description gives enough detail for an agent to understand the tool's purpose, inputs, and outputs, including severity and rotation status. It lacks explicit failure modes and usage alternatives, but for a read-only scan tool with a well-described schema, it is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds context around 'recent commits' which relates to since_days and last_n_commits but does not explain defaults or how the two parameters interact, adding only marginal value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Scan'), specifies the resource ('git commit history of a repository'), and states the exact purpose ('for secrets that were ever committed, even if later deleted'). This clearly distinguishes it from siblings like scan_for_secrets by emphasizing historical commit detection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description sets clear context for when to use the tool (scanning git history for secrets, including deleted ones) but does not explicitly mention alternatives or when not to use it. The sibling differentiation is implied through 'even if later deleted' rather than stated directly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.