Server Quality Checklist

Tool Scores

• lint_codeC
  2.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided. The description only mentions the return shape ({ok, parse_error, findings}) but omits any behavioral traits such as whether it is read-only, required permissions, side effects, or rate limits. The burden is on the description to disclose such behaviors, and it fails to do so.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short (one sentence plus return type), which is concise. However, it is too brief, missing important details. Every sentence does earn its place, but the description could be more informative without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With two parameters (one optional), no parameter descriptions, no annotations, and an output schema only hinted at in the description, the description is critically incomplete. It does not explain what 'leakage patterns' means, how to interpret the return, or any context about the tool's behavior. This is insufficient for proper tool selection and use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not elaborate on the parameters. 'code' is implied as the source string, but 'filename' is optional and has no explanation. The description provides no additional meaning beyond what the schema gives, which is minimal.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scans a source string for leakage patterns, which is a specific verb+resource combination. It implicitly distinguishes from siblings like lint_file and lint_paths that operate on files or paths. However, it does not explicitly contrast with these siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description only implies it is for scanning a source string, but does not mention when to choose lint_code over lint_file or lint_paths, nor any prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lint_fileA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the return shape ({ok, parse_error, findings} or error dict) but omits details like error handling for missing files or performance characteristics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. It could integrate sibling differentiation without added length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one param, output schema exists), the description covers the basic purpose and return format but lacks guidance on error states and prerequisite conditions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The sole parameter 'path' is a string with 0% schema coverage. The description adds only that it's a file on disk, failing to clarify path format (absolute/relative) or constraints beyond type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Scan a single file on disk' with a clear verb and resource, and distinguishes from siblings like lint_code and lint_paths through the 'single file' scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for single files on disk but does not explicitly contrast with alternatives such as lint_code for code strings or lint_paths for multiple files.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_ruleA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the tool returns 'long-form rationale' and 'canonical fix pattern', indicating a read-only information retrieval. However, it does not describe output format, latency, or any prerequisites (e.g., rule_id must exist). The description is adequate but lacks depth for a fully transparent behavioral profile.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that efficiently conveys the tool's purpose, scope (one rule id), and output type (long-form rationale and fix pattern). No unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has only one parameter, no annotations, and an output schema (not shown but indicated), the description covers the core function without major gaps. It could be improved by clarifying that the rule_id must be valid and that the output is textual, but it is largely complete for its simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one parameter 'rule_id' with no description. The tool description provides an example ('e.g. 'LG001''), which adds some meaning by hinting at the format (likely rule identifiers like those in sibling tools). However, with 0% schema description coverage, the description does not fully explain the parameter's validation or accepted values. The example is useful but not exhaustive.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool provides 'long-form rationale and canonical fix pattern for one rule id', specifying the verb 'explain' (implied by the name) and the resource (a specific rule id). It distinguishes from sibling tools like 'list_rules' which lists all rules, by focusing on a single rule id.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for getting detailed rationale and fix pattern for a specific rule id, but it lacks explicit guidance on when to use this tool versus alternatives like 'lint_code' (which might apply rules) or 'list_rules' (which lists all rules). No when-not-to-use or alternative conditions are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lint_pathsA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes the core action (scan every file matching a glob) and return shape. While no annotations exist, the description implies a read-only operation without side effects. Could mention potential resource constraints or glob evaluation behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences front-loading the action and return type. No extraneous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one parameter and an output schema, the description adequately covers the purpose and return shape. Missing details like glob syntax or edge cases, but still functional for its simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description only says 'a glob' for the pattern parameter. Does not specify glob syntax, workspace location, or any constraints; relies on user understanding of glob patterns.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it scans files matching a glob pattern and returns a structured result with ok/parse_error/findings per path. Distinct from sibling tools that operate on single files or code snippets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage for batch scanning via glob, but no explicit guidance on when to use this tool vs. alternatives like lint_file or lint_code. No exclusion criteria or suggested use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_rulesA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden but only states the output fields. No disclosure of side effects, auth needs, or rate limits, though for a read-only list tool this is minimal risk.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence that is direct and front-loaded, conveying exactly what the tool returns. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and an output schema, the description adequately covers the tool's behavior. The agent can infer the rest from the output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so schema coverage is 100%. Baseline 4 applies; description adds no parameter info but none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it provides a 'full rule catalog' with specific fields (id, name, severity, one-line summary). It distinguishes itself from siblings like explain_rule (explanation) and lint tools (analysis).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit when/when-not guidance or alternatives to siblings. However, given the tool's simplicity and distinct sibling purposes, usage is implied but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.