Server Quality Checklist

Tool Scores

• audit_ci_securityA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavioral traits: it performs heuristic static analysis, requires an API key for deep audit, redacts secrets, returns a verdict and SARIF report, and lists many categories of issues. It also mentions limitations ('not a guarantee').

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is excessively long (wall of text). While it has structure (ecosystem-by-ecosystem), it contains unnecessary details that could be shortened or placed in schema descriptions. The verbosity reduces clarity for an AI agent trying to parse key points quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 ecosystems, deep audit modes, API key requirement, return format), the description is comprehensive. It explains inputs, operations, outputs (verdict and SARIF report), and even covers edge cases like cross-file analysis and secret redaction.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, yet the description adds significant value beyond the schema. For 'deep,' it explains the need for an API key. For 'files' and 'source,' it adds details about auto-detection and mixing ecosystems. For 'sharedLibYmls,' it explains cross-file taint analysis, which is not in schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool audits CI configs for security flaws, listing seven specific CI ecosystems. The verb 'Audit' and resource 'CI config' are explicit, and the tool is distinguished from its sibling 'diff_ci_security' by focusing on auditing rather than diffing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises using the tool 'whenever reviewing, writing or accepting CI config,' providing clear context for its use. However, it does not explicitly state when not to use it or mention the sibling 'diff_ci_security' as an alternative for diffing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diff_ci_securityA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: it runs the full engine on both states, reconciles findings using line-independent identity, produces verdicts, and provides detailed findings. It also notes it uses 'heuristic static analysis, not a guarantee' and mentions premium requirements (API key). This is thorough and transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is quite long but every sentence adds value. It is front-loaded with purpose and usage. While it could be slightly more concise, it avoids redundancy and maintains a clear logical flow.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (2 required params with nested objects, no output schema), the description is remarkably complete. It explains input format, output verdicts, and the line-independent identity mechanism. It even describes the findings' structure (file:line, taint path, fix). All essential information is covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the description adds substantial meaning: it explains the before/after shape, mentions the two formats (files/source), lists additional optional fields like orbYmls and actionYmls, and clarifies how to provide each side. This goes beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it performs a 'DIFFERENTIAL CI/CD security check', comparing before/after states of CI config files. It explicitly distinguishes from the sibling tool 'audit_ci_security' which likely does single-state auditing. The verb and resource are specific, and the purpose is unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use: 'Use it on every PR that touches CI config — wire INTRODUCES_RISK to a failing status check.' It contrasts with single-state analysis, explaining that this tool provides a delta that an agent cannot compute alone. This gives clear context and alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.