Server Quality Checklist

Tool Scores

• list_semantic_changesA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description must carry the burden. It states 'Lists all semantic changes' suggesting read-only behavior, but lacks details on return structure, pagination, side effects, or error conditions. Incomplete for a tool with no annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with action and examples. No wasted words, efficient and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 parameters, no output schema, and no annotations, the description explains purpose and examples but lacks details on return format, error handling, or performance. Adequate but with gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (3 parameters all described). The description adds little beyond the schema: it mentions 'relative to a base' which aligns with baseBranch, but no deeper semantics. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool lists semantic changes on a branch relative to a base, with specific examples like function additions/removals, export changes, type changes. Distinguishes itself from sibling tools (analyze_branches, analyze_file_pair, check_merge_safety) which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description says 'Useful for understanding what semantically changed,' implying when to use it, but no explicit guidance on alternatives or when not to use it. No exclusions or comparisons with sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_file_pairA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden. It discloses the tool 'shows' results, implying a read-only operation, but does not explicitly state it has no side effects or any performance considerations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is a single sentence that is direct and front-loaded with the purpose. While concise, a second sentence on usage or output could improve completeness without harming conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and 4 parameters, the description explains the tool's purpose and high-level output (changes and conflicts), but lacks details on return format or how results are presented, which is important for a diff tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents all parameters. The description adds no extra meaning beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs a 'deep-dive semantic diff of a single file between two branches' and shows changes and conflicts. This distinguishes it from siblings like analyze_branches which likely covers multiple files.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool vs alternatives (e.g., analyze_branches for broader diffs). The context is implied but not stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_branchesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description provides some behavioral context (returns detailed report, includes conflicts and cross-file issues), but lacks details on potential side effects, performance, or required permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: one stating purpose and one describing output. No unnecessary words; efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description adequately explains the return value (per-file report of changes and conflicts). However, it could mention prerequisites like branch existence or edge cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, and the description does not add new meaning beyond the schema's parameter descriptions. It only reiterates the general purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it performs deep semantic analysis between two branches and returns a per-file report of changes and conflicts, distinguishing it from sibling tools like 'analyze_file_pair' and 'check_merge_safety'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for semantic comparison but does not provide explicit guidance on when to use this tool versus alternatives like 'list_semantic_changes' or 'check_merge_safety'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_merge_safetyA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must bear the full burden. It discloses the return risk score (safe/warning/danger), but does not mention side effects (e.g., read-only nature) or execution time. Adequate but could add more detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, concise and well-structured. No unnecessary words, and the key information is front-loaded (quick pre-merge safety check).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 3 required parameters and no output schema, the description provides a clear purpose and return risk score. It lacks details on the exact format of the risk score or any additional output, but is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (all three parameters have descriptions). The description adds that the tool analyzes branches for semantic conflicts, but this is implied by the overall purpose and does not significantly enhance parameter understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: a quick pre-merge safety check analyzing two branches for semantic conflicts. It specifies what it catches (signature changes, etc.) and distinguishes it from sibling tools like analyze_branches and list_semantic_changes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use before merging by stating 'Quick pre-merge safety check'. It mentions conflicts Git would miss, but does not explicitly state when not to use it or compare with alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.