Server Quality Checklist

Tool Scores

• AuditCodeIntegrityA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description must disclose safety and side effects. It mentions returning structured JSON findings and lists checks, but does not state whether the tool is read-only, whether it modifies files, or any permission requirements. Behavioral details beyond the basic function are absent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with the verb and resource first, followed by a compact parenthetical list of checks. It provides all essential information without wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema or annotations, the description should compensate by explaining the return structure or behavioral constraints. It only says 'structured JSON findings', which is vague. The check list adds context but does not fully prepare the agent for expected outputs or side effects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage for the two parameters is 100%, so the description adds no significant meaning beyond the schema. The 'file or directory' phrase mirrors the target description, and no additional syntax or parameter-specific guidance is provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Run' and identifies the resource as 'code integrity auditor', listing concrete check categories (lifecycle teardown parity, mock-theater test detection, naming invariants, swallowed exceptions) and targets (file or directory). This clearly distinguishes it from the sibling safety checkers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for auditing code integrity but provides no explicit when-to-use guidance, alternatives, or exclusions relative to sibling tools like JsSafetyChecker. It only states the input target, not when to choose this tool over others.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ShellSafetyCheckerA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the burden. It discloses the specific checks and states 'Returns findings,' which implies a read-only analysis. However, it does not explicitly state non-destructive behavior, permission requirements, or failure modes, which is a gap given zero annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the action, packs the key checks into a parenthetical, and concludes with the outcome. No wasted words or redundant content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's scope and checks, but since there is no output schema, the vague 'Returns findings' fails to explain the return format or structure. For a checker tool with zero annotations and output schema, this leaves a notable gap in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides full descriptions for both parameters (target and timeout), and the description adds no extra parameter semantics. Baseline 3 is appropriate because the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Run the shell safety checker'), the resource ('shell scripts'), and enumerates the specific checks performed. The mention of shell scripts distinguishes it from sibling JS/Python safety checkers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'against shell scripts,' which clearly implies when to use this tool over the JS/Python siblings. However, it does not explicitly name alternatives or provide exclusion criteria, so it stops short of full guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• JsSafetyCheckerA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavior. It mentions the type of analysis (AST-based) and that it returns findings, which suggests a non-mutating, read-only check. However, it does not explicitly state that it does not modify files, nor does it describe any side effects, rate limits, or other behavioral nuances.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys the tool's purpose, method, target, and output in a compact form. Every word is informative and there is no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, with two well-documented parameters and no output schema. The description says 'Returns findings' but does not specify the structure or format of the findings, which is a minor gap given no output schema exists. Otherwise, it is complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage with descriptions for both parameters (target and timeout). The description only repeats the 'file or directory' target scope, adding no significant meaning beyond the schema. This meets the baseline but does not elevate it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Run') and clearly identifies the resource (JavaScript safety checker) and its specific AST-based checks (duplicate function, duplicate HTML ID, syntax errors). This distinguishes it from sibling tools like PythonSafetyChecker and ShellSafetyChecker.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies that this tool is for scanning JavaScript files/directories, which gives clear context. However, it does not explicitly mention when not to use it or point to alternatives (e.g., 'use PythonSafetyChecker for Python'), so it lacks explicit exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• PythonSafetyCheckerA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states that it 'runs' the checker and 'returns findings', which implies a read-only analysis, but it does not explicitly say it does not modify files. It also does not describe the output format or potential error conditions. It does add useful context about the specific checks performed, but leaves ambiguity about non-destructive behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, consisting of two sentences that are front-loaded with the tool's purpose. Every piece of information (the types of checks and the fact that findings are returned) earns its place. There is no redundant or filler content, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (two optional params, no required params, no output schema), the description is sufficiently complete for an agent to understand what it does and what to expect. It covers the purpose and the high-level result ('Returns findings'), though it does not detail the findings' structure. Annotations are absent, but the description compensates with specifics about the checks performed. It is adequate for selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers both parameters (target, timeout) with descriptions, achieving 100% coverage. The tool description does not add any additional meaning beyond the schema; it only mentions the target implicitly through 'against Python files'. Since the schema already explains the parameters, the description provides no extra semantic value, aligning with the baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a Python safety checker, specifying the exact checks (Bandit security, Ruff style/lint, AST mutable default args) and that it targets Python files. The verb 'Run' specifies the action, and the resource is the Python checker. This effectively distinguishes it from sibling tools that target other languages.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for Python files, which is the clear context. It does not explicitly state when not to use it or name alternatives, but the sibling tools (JsSafetyChecker, ShellSafetyChecker) and the direct mention of 'Python files' provide enough situational guidance. There are no exclusions or prerequisites, making it a clear context without full when/when-not articulation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.