Server Quality Checklist

Tool Scores

• explain_issueC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool returns explanations and guidance, but doesn't describe behavioral traits like whether it's read-only, if it has side effects, rate limits, or authentication needs. For a tool with no annotations, this leaves significant gaps in understanding how it behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, stating the core purpose in the first clause. It uses two sentences efficiently to cover what the tool does and what it returns. There's no wasted verbiage, though it could be slightly more structured by explicitly separating purpose from output.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (6 parameters, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose and return types but lacks details on behavioral traits, usage context relative to siblings, and output format specifics. With no output schema, the description should ideally explain return values more thoroughly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description doesn't add any parameter-specific information beyond what's in the input schema. Since schema description coverage is 100%, the schema already documents all 6 parameters thoroughly. The baseline score of 3 is appropriate as the description doesn't compensate but also doesn't detract from the comprehensive schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Explain a code quality issue detected by OCR.' It specifies the verb (explain) and resource (code quality issue), and mentions the return content (detailed explanation, category context, fix guidance). However, it doesn't explicitly differentiate from sibling tools like 'heal_code' or 'scan_diff' which might handle similar issues.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal usage guidance. It implies this tool should be used when an AI agent needs to understand and act on a code quality issue, but it doesn't specify when to use this versus alternatives like 'heal_code' (which might fix issues) or 'scan_diff' (which might detect them). No explicit when/when-not instructions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heal_codeC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the tool loads code and prepares a repair prompt, but lacks details on behavioral traits such as permissions needed, whether it modifies the file (implied by 'heal' but not confirmed), error handling, rate limits, or what 'repair context' entails. This leaves significant gaps for a tool with potential mutation implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, stating the core action in the first sentence. Both sentences earn their place by explaining the tool's function and the agent's role. However, it could be slightly more structured by explicitly separating tool behavior from agent instructions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete. It mentions returns 'file content along with the repair context' but doesn't detail the output format or behavioral aspects like side effects. For a tool named 'heal_code' with potential mutations, more context on safety, response structure, and error cases is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents parameters (path, issue, suggestion). The description adds marginal value by implying parameters are used to 'prepare a repair prompt' and 'apply the fix,' but doesn't provide additional syntax, format, or usage details beyond what the schema already specifies. Baseline 3 is appropriate here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Load a file's source code and prepare a repair prompt for the AI agent.' It specifies the verb ('load' and 'prepare'), resource ('file's source code'), and outcome ('repair prompt'). However, it doesn't explicitly differentiate from sibling tools like 'explain_issue' or 'scan_diff', which might also involve code analysis or repair contexts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal guidance on when to use this tool. It mentions the agent should 'apply the fix based on the issue description and suggestion,' implying usage for code repair scenarios, but offers no explicit when-to-use vs. alternatives, prerequisites, or exclusions compared to siblings like 'scan_directory' or 'explain_issue'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_directoryB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only partially discloses behavioral traits. It mentions detection capabilities and language support, but omits critical details like whether the scan is read-only, its performance impact, error handling, or output format. For a tool with no annotations, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core action and key detection categories. Both sentences earn their place by specifying scope and capabilities, though it could be slightly more structured by separating usage notes from feature lists.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (scanning for multiple issue types across languages) and lack of annotations and output schema, the description is incomplete. It doesn't explain what the scan returns, how results are formatted, or any behavioral constraints like rate limits or permissions needed, making it inadequate for full contextual understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters well. The description adds minimal value beyond the schema by mentioning language support, which aligns with the 'languages' parameter, but doesn't provide additional syntax or format details. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('scan', 'detects') and resources ('directory', 'AI-generated code quality issues'), listing concrete detection categories like hallucinated imports and security anti-patterns. It distinguishes from sibling tools by focusing on scanning rather than explaining or healing issues.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for scanning directories in supported languages, but lacks explicit guidance on when to use this tool versus alternatives like 'scan_diff' or when not to use it. No prerequisites or exclusions are mentioned, leaving usage context somewhat vague.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_diffA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool 'only analyzes changed files and lines,' which is useful behavioral context about its scope. However, it doesn't mention performance characteristics (e.g., speed, resource usage), error handling, or what constitutes 'code quality issues,' leaving gaps in behavioral understanding.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with zero waste. The first sentence states the core purpose, and the second adds crucial context about scope and ideal use case. Every word earns its place, and information is front-loaded effectively.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description provides adequate purpose and usage context but lacks details on what the tool returns (e.g., issue list format, severity levels) or behavioral traits like error conditions. For a tool with 4 parameters and no structured output documentation, this leaves room for improvement in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds no additional parameter semantics beyond what's in the schema (e.g., it doesn't explain what 'SLA level' means or provide examples for 'path'). Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('scan'), resource ('git diff between two branches'), and target ('for code quality issues'). It distinguishes from sibling tools like 'scan_directory' by specifying it only analyzes changed files and lines, making it ideal for PR/MR review contexts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('Ideal for PR/MR review') and specifies it 'only analyzes changed files and lines,' which implicitly differentiates it from 'scan_directory' that likely scans entire directories. However, it doesn't explicitly state when not to use it or name alternatives like 'explain_issue' or 'heal_code' for related tasks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.