Server Quality Checklist

Tool Scores

• generate_changelogC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must fully disclose behavioral traits, but it only states it 'generates a changelog'. It does not mention if the tool is read-only, requires specific permissions, or what 'categorized' entails.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, which is efficient, but it lacks any structure (e.g., bullet points) and omits critical details, making it overly brief.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 3 parameters, no schema description coverage, and an output schema exists but is not described, the description is severely incomplete. It provides no guidance on usage, behavior, or parameter semantics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description provides no details about the parameters base, revision, or output_format. It fails to explain their meaning, expected format (e.g., file paths or raw content), or allowed values for output_format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the verb 'generate', the resource 'categorized changelog', and the scope 'between two OpenAPI documents'. This effectively distinguishes it from sibling tools like validate_spec, compare_specs, and list_breaking_changes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives, such as compare_specs or list_breaking_changes. There is no mention of context, prerequisites, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_specsC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden of behavioral disclosure. It only states the action but omits details about authentication, rate limits, side effects, or the nature of the diff (e.g., read-only). The minimal transparency leaves critical behavioral traits unaddressed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no waste. It is front-loaded with the primary action. However, it could be restructured to include key details without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (3 parameters, 2 required, no output schema shown despite presence), the description is insufficient. It fails to specify return format, parameter constraints, or how this tool differs from siblings. The lack of parameter and output documentation leaves the agent underinformed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description adds no explanation for the parameters 'base', 'revision', or 'output_format'. The agent cannot infer input format (e.g., file path vs. URL) or the meaning of 'output_format' options beyond the default 'text'. This is a critical gap given the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Compare two OpenAPI documents and return the complete structural diff,' which specifies the verb (compare) and resource (OpenAPI documents) and distinguishes this tool from siblings like validate_spec, list_breaking_changes, and generate_changelog.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives. It does not mention prerequisites, when-not-to-use, or refer to sibling tools, leaving the agent to infer usage from the purpose alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_breaking_changesC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes the basic output (breaking changes) but lacks details on behavior such as whether it returns a list, error scenarios, or if output_format affects content. Since no annotations are provided, the description carries full burden and is moderately transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence is very concise but too brief, lacking structure and detail. It front-loads the purpose but omits necessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 3 parameters and no schema descriptions, the description is incomplete. It does not explain parameters or output details, leaving the agent underinformed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description makes no mention of parameters. The agent has no clues about what 'base' and 'revision' represent, nor the options for 'output_format'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's action as listing breaking changes in a revision. It distinguishes from sibling tools: validate_spec (validation), compare_specs (comparison), and generate_changelog (full changelog).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use or not use this tool versus siblings. No context on prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_specA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description effectively discloses the key behavioral trait of not resolving external references, which sets expectations for validation scope. It does not explicitly state that the tool is read-only, but that is implicit for a validator.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that conveys all essential information without redundancy. Every word serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema present), the description provides sufficient context for the agent to understand its behavior and constraints. It could mention that it returns validation results, but the output schema likely covers that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning to the single parameter 'source' by indicating it refers to a local OpenAPI 3.x file, but it does not specify whether it expects a file path or file content. With 0% schema coverage, the description partially compensates, but more detail on input format would help.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Validate'), the resource ('a local OpenAPI 3.x file'), and a key constraint ('without resolving external references'), making the tool's purpose distinct from siblings like compare_specs or list_breaking_changes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (validating a local spec file) but does not explicitly contrast with sibling tools or mention when not to use. It could be improved by specifying that this tool is for local files only and does not compare or generate changelogs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.