Server Quality Checklist

Tool Scores

• match_rulesB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full responsibility for behavioral disclosure. It mentions returning results and merging, but does not address read-only nature, error handling (e.g., file path not found), authentication, or performance constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 20 words, efficiently communicating the core functionality without extraneous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with two parameters and no output schema, the description covers the primary purpose. However, it lacks usage guidelines and behavioral details, which would enhance completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents both parameters. The description mentions 'file path' aligning with the 'path' parameter but does not elaborate on 'repo' or add new meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns applicable review rules for a file path and merges them into a prompt section. It uses a specific verb (Return, merged) and resource (review rules, prompt_section), distinguishing it from sibling tools like get_file_bundle or get_review_targets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor does it mention prerequisites or when not to use it. Sibling tools are listed but not referenced.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• position_commentB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explains the matching algorithm (text matching primary with hunk alignment fallback) and references another tool, giving some insight into behavior. However, with no annotations, it fails to disclose safety aspects (e.g., whether it is read-only or modifies state), which is important for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief (two sentences) and front-loads the purpose. However, the second sentence is somewhat technical and could be simplified for broader understanding without losing meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 7 parameters, no output schema, and no annotations, the description covers the core positioning logic but omits details on error cases, prerequisites, and what the output looks like. It is adequate but not fully comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 86%, so most parameters are already documented in the schema. The description adds context only for 'diff_ref' by referencing its source. The parameter 'path' lacks a schema description, and the tool description does not compensate for that, leaving it partially unclear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the verb 'locate' and specifies the resource 'comment' with the goal of precise line numbers, making the tool's primary function clear. However, it does not explicitly distinguish it from the sibling 'reflect_comment' tool, which may have overlapping functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a specific usage hint to 'Pass diff_ref from get_review_targets', which helps with integration. However, it does not specify when to use this tool versus alternatives like 'reflect_comment', nor does it give conditions for when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_review_targetsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must convey behavioral traits. It states 'Applies file filtering' but does not explain what filtering is applied (e.g., based on file extensions or rules). It also mentions returning a diff_ref but does not describe the format or any side effects. The description is adequate but lacks detail on how results are determined.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two sentences that efficiently convey purpose and key behavior. Every word adds value, with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters (some conditional) and no output schema, the description is moderately complete. It omits details like how mode affects parameter requirements and what the diff_ref actually contains. However, the schema fills in many gaps. For a tool that feeds into downstream tools, it may be sufficient, but it leaves some ambiguity for standalone use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 5 parameters are described in the input schema with full coverage (100%). The description adds no additional insight beyond the schema, e.g., it does not explain how 'mode' controls filtering or that 'to'/'from' are conditional. With high schema coverage, baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Determine which files need review from a git diff.' It specifies the verb (determine), resource (files from a git diff), and mentions the output ('returns a diff_ref to pass to downstream tools'). It also hints at its role in a pipeline, distinguishing it from siblings like get_file_bundle or position_comment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have a git diff and need to filter files for review, but it does not explicitly state when to use this tool versus alternatives. No guidance on prerequisites, when not to use, or mention of sibling tools like get_file_bundle for bundling files.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reflect_commentA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It explicitly states the tool is deterministic and lists the three checks (line_in_hunk, existing_code_found, existing_code_in_diff), providing good transparency. However, it does not mention side effects, authorization requirements, or error handling, which are minor gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences long, front-loaded with purpose, and every sentence adds value. No fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters and no output schema, the description covers the core logic (three checks) but leaves gaps: it doesn't explain what each check means, the exact return format beyond 'keep or drop', or error conditions. More detail on outcomes would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 71%, so the description's additional value is limited. It mentions 'positioned comment' and the three checks, but does not explicitly map parameters to those checks. The schema descriptions cover most parameter meanings, so the description adds only context about the checks.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Deterministic validation of a positioned comment' and lists the three specific checks, clearly defining the tool's function and output ('keep or drop'). It distinguishes itself from sibling tools like position_comment or get_review_targets by focusing on validation rather than creation or retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes that the tool 'Does not call LLM', implying it is a lightweight alternative, but does not explicitly state when to use this tool versus siblings like position_comment or match_rules. The context of three checks suggests it is for validating a comment after positioning, but no clear when-to-use or when-not-to-use guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_file_bundleA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses the 20000 char cap, a key behavioral constraint. Mentions dependency on get_review_targets, implying data flow. No annotations provided, so description carries the burden well. Could mention whether it modifies state, but mutation is unlikely.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences covering purpose, examples, constraint, and usage hint. No fluff, front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequately describes the tool's behavior and dependencies for a 3-parameter tool with no output schema. References sibling tool. Slight gap: no mention of return value or side effects, but not critical for a grouping operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds context for diff_ref ('pass from get_review_targets') but files parameter repeats schema description. Some added value but not critical.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the verb 'Group' and resource 'files into review bundles' with specific examples (test/source pairs, i18n variants) and a constraint (20000 char cap). Distinguishes from sibling get_review_targets by indicating that tool provides input.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to pass diff_ref from get_review_targets, giving clear usage context. Does not explicitly exclude other scenarios, but the instruction is strong enough for most use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.