Server Quality Checklist

Tool Scores

• generate_migration_guideA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false. The description adds clarity by stating it 'analyzes API contract compatibility only' and 'does NOT modify code', which reinforces the non-destructive nature and provides additional behavioral context beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the key purpose, and contains no unnecessary words. Every sentence contributes to understanding the tool's function and constraints.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there is no output schema, the description conveys that the output is a markdown guide and safe rollout steps. It also clarifies the scope (API contract compatibility only). This is sufficient for a tool with moderate complexity. A higher score would require more detail on parameter relationships or output structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is high (80%), so the baseline is 3. The description does not add extra meaning to the parameters; it does not explain when to use file paths versus inline content. The language enum is mentioned but not elaborated. The description adds marginal value over the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool generates a migration guide (markdown) and safe rollout steps for breaking changes in an API spec change. It distinguishes itself from siblings like compare_specs or explain_breaking_changes by specifying the output format and the focus on safe rollout steps.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool versus its siblings. It does not mention prerequisites, alternatives, or when not to use it, leaving the agent to infer usage context from the purpose alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• is_change_safeA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, indicating a safe, read-only operation. The description adds value by stating it analyzes API contract compatibility only and returns a verdict, risk level, reasons, and action. It does not contradict annotations and provides useful context beyond them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise, front-loading the core purpose ('THE DEPLOY GATE') and then succinctly covering what it does, what it returns, when to use it, and what it does not do. Every sentence serves a clear function with no waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains the return values (verdict, risk level, reasons, action) adequately since there is no output schema. The parameters are well-documented in the schema. Sibling tools are listed but not explicitly differentiated. Overall, the context is complete for the tool's intended use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 6 parameters with 83% description coverage, so the schema already documents most parameters well. The tool description does not add additional meaning or syntax details beyond what the schema provides. Baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it is a deploy gate that analyzes API spec changes for safety, and it lists the specific outputs (safeToMerge verdict, risk level, etc.). However, it does not explicitly differentiate itself from siblings like compare_specs or explain_breaking_changes, which could cause confusion about when to use this tool over others.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this before merging or releasing an API change,' providing clear when-to-use guidance. It also states it does not modify code, implying a read-only analysis. However, it does not mention when not to use it or alternative tools for different needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_breaking_changesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint and non-destructive. The description adds that the tool analyzes compatibility only and returns impact and migration steps, providing useful context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose, no wasted words. Concisely conveys what the tool does and its constraints.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains return content (developer impact, consumer impact, migration steps). It could mention input constraints like required parameter combinations, but overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and each parameter has a clear description. The description does not add additional meaning beyond what the schema provides, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Explains what breaks in an API spec change and why' and lists specific outputs. It also distinguishes from siblings by noting it does not modify code.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for analyzing API contract compatibility and explicitly states it does not modify code, but lacks explicit guidance on when to choose this over siblings like compare_specs or generate_migration_guide.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_release_notesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true and destructiveHint=false. Description reinforces no code modification and adds scope (API contracts only). This is consistent and adds context beyond annotations, though it could mention openWorldHint (e.g., no side effects besides reading).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence of 29 words, front-loaded with action and outcome. No unnecessary details, every part adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only tool with 5 parameters, high schema coverage, and no output schema, the description adequately explains output (markdown, grouped) and purpose. It could mention that output is returned as markdown string, but overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 80% (4/5 parameters documented). The description does not add further explanation beyond what the schema provides, such as clarifying the 'audience' enum values. Baseline 3 is appropriate as schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool generates release notes in markdown from API spec changes, grouped into breaking/additions/other, tailored to audience. It explicitly says it does not modify code, distinguishing it from sibling tools like compare_specs or generate_migration_guide.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description states it analyzes API contract compatibility only and does not modify code, providing implicit guidance on when to use (for release notes) and when not (for actual code changes). However, it does not explicitly name alternative tools or provide when-not-to-use conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_specsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false. The description adds that the tool 'does NOT modify code' and 'analyzes API contract compatibility only,' which is consistent but slightly redundant. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first defines the function, second gives usage guidance. No wasted words, front-loaded with key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description lists the output components (breaking changes, additions, modifications, warnings, risk score, compatibility summary). Parameters are well-documented in the input schema. All necessary context is present for correct tool invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 83% (5 of 6 parameters have descriptions). The tool description does not add any additional parameter details beyond what the schema provides, so it meets the baseline for high coverage but does not enhance understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool compares two API specs and lists breaking changes, additions, modifications, and warnings with a risk score and compatibility summary. It also distinguishes from sibling tool 'is_change_safe' by noting that for merge/deploy decisions the latter is preferred.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear guidance: 'For a merge/deploy decision prefer is_change_safe.' This tells the agent when to use this tool versus an alternative, and it also clarifies that the tool analyzes but does not modify code.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.