Server Quality Checklist

Tool Scores

• validate_responseC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are absent, so the description carries the full burden. It tells the agent what the tool does but not what happens on failure (e.g., thrown error vs return value), whether the spec must already be loaded, or whether this mutates any state. The description is merely a restatement of the tool's name, adding little beyond the title.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence, no fluff, front-loaded with the action. It earns its place by stating the tool's purpose succinctly, though it could add more detail without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations and no output schema, a complete description should explain error behavior, whether the schema is loaded internally or requires prior loading, and how to interpret the result. None of this is present. The description is minimal and leaves too much to agent inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, and the description does not describe any parameter behavior. All four parameters (path, method, status, responseBody) are required, but nothing in the description explains their format, e.g., whether responseBody is raw text or an object, or whether status must be an HTTP numeric code. The description does not compensate for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear verb (check), a specific resource (response body), and a criterion (JSON schema for method + path + status). It is distinguishable from siblings like call_endpoint and list_endpoints, but the title and description are nearly synonymous. It does not name sibling alternatives or any exclusions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus alternatives. The description is contextual but not directive; an agent cannot tell whether to call this versus load_api_spec or list_endpoints. There are no explicit exclusions or alternative routing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• call_endpointA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden. It does disclose two important traits: the call is real/actual and the response is not validated. However, it does not mention that this can trigger mutating side effects (e.g. DELETE or POST), whether auth is needed or applied automatically, what happens on network errors or non-2xx statuses, or whether an API spec must be loaded first. For a tool that makes real network calls, these are material gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two focused sentences with no filler. The primary behavior and return payload are front-loaded, and the key distinction from test_endpoint is given immediately afterward. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity — 9 parameters, a nested auth object, no output schema, and no annotations — this description is too thin. It covers the basic response characteristics and one sibling distinction, but leaves the agent without guidance on constructing path/openAPI requests, handling auth, avoiding accidental destructive calls, or interpreting failures. The schema partially fills some gaps, but the overall contextual picture is incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 56%, and the description adds no parameter-level meaning. It does not explain how to supply pathParams versus queryParams, how the auth object works, what body should contain, or how headers are applied. The missing details for body, headers, pathParams, and queryParams are not compensated by the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the exact action ('Make an actual HTTP call') and the exact resource ('a documented endpoint'), and explicitly lists the return values: real status, headers, and body. It distinguishes itself from the sibling test_endpoint by stating it does not validate responses, so an agent can tell them apart immediately.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It names the direct alternative, test_endpoint, and gives the condition: use test_endpoint when schema validation is also desired. This is an explicit when-not/alternative instruction. Other siblings serve clearly different purposes such as loading specs or running test suites, so the key routing decision is covered.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• test_endpointA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It does disclose the core behavior—sending a real request, validating the response, returning pass/fail and mismatches. However, it does not mention side effects of real calls, authentication requirements, or failure modes, which are relevant for a no-annotation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short sentences, each earning its place: action, return value, and primary use. The key behavior is front-loaded and there is no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations, no output schema, and 9 parameters, the description lacks operational context: how the documented schema is made available, what happens on non-JSON or network errors, and how the returned mismatches are formatted. The pass/fail behavior is stated, but not enough for fully correct invocation in a complex workflow.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 9 parameters with only about 22% description coverage, so the description needed to compensate. It only says 'with real parameters' and gives no guidance on method/path, auth, body, or path/query parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific combined operation—call an endpoint and validate it against its documented schema—and positions it as 'the main tool for does this endpoint actually work as documented'. This clearly differentiates it from siblings like call_endpoint or validate_response.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says when to use the tool: whenever you need to know whether an endpoint works as documented. It does not spell out exclusions or direct to alternatives such as call_endpoint for raw calling or validate_response for already-captured responses, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_healthA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It transparently states that the tool makes live calls, is one-shot, defaults to parameter-free GET endpoints, and reports status/latency. However, it does not warn about potential side effects when explicit non-GET endpoints are provided, nor about prerequisites like a loaded spec or auth setup.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core behavior and ending with concrete use cases. Every clause contributes, and there is no redundant filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema, the description usefully enumerates the key output dimensions (responded, status code, latency) and explains default endpoint selection. It does not cover edge cases like no spec loaded or network failures, but those are minor for a health-check tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is high (~75%) and already documents endpoints, timeout, and auth. The description reinforces the default endpoint behavior and the reported output fields, but it does not add substantial parameter-level meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb/resource pairing: a one-shot health check that calls endpoints and reports whether they responded, with status code and latency. It is fully clear about what the tool does, but it does not explicitly contrast itself with sibling tools like test_endpoint or run_all_tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context: it is positioned as a quick 'is the API up' pass, e.g., before a demo or in CI. It does not state when not to use it or point users to alternatives, so it stops short of full guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• load_api_specA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It does disclose key behavior: full dereferencing, JSON/YAML support, and that it returns title, servers, and every endpoint. However, it does not mention error handling, network dependency, or whether dereferencing can resolve remote references, leaving some behavioral uncertainty.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences with no filler. The primary action and return value are front-loaded, and the usage instruction 'Call this first' is placed early. Every phrase earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter loading tool with no output schema, the description tells the agent what it returns and when to call it. It is reasonably complete, though error or dereferencing edge cases are not covered. Those gaps are minor for the agent's selection and invocation decision.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides complete coverage of the single 'source' parameter with a clear example. The description adds 'JSON or YAML' and 'fully dereference' context, but this is only marginal value beyond the schema. Baseline 3 is appropriate given high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Load and fully dereference an OpenAPI/Swagger spec from a URL or local file path.' It also clarifies what the tool returns, immediately distinguishing it from sibling tools like list_endpoints, call_endpoint, and test_endpoint. This makes it unambiguous as the initial loading step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'Call this first' explicitly signals prerequisite usage before other API tools. It does not enumerate when-not-to-use or name alternative loaders, but the sibling list makes the workflow context clear. This is strong practical guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diff_api_specsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the behavioral disclosure. It does so by stating what it reports and by explicitly guaranteeing no side effects on the currently loaded spec ('Does not affect the spec currently loaded for the other tools'). It does not mention network/error behavior, but the core behavior and safety profile are clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no filler. The first sentence packs the core action, input types, example, and output classification; the second sentence adds the crucial side-effect guarantee. All content earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description tells the agent what the tool returns conceptually: added/removed endpoints, request/response schema changes, and a distinction between breaking and safe changes. Combined with the input format and side-effect note, an agent can decide when and how to invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters have schema descriptions ('The before spec', 'The after spec'), so schema coverage is 100%. The description adds the valid input type (URL or local file path) and illustrates intended usage with the old tag vs current main example, which is meaningful beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific action ('Compare two OpenAPI/Swagger specs'), a clear subject (the two specs), and the concrete output (added/removed endpoints, schema changes, breaking vs. safe). This differentiates it from siblings that load, list, call, or validate against a single spec.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context: use this when you need to compare an old version (e.g., old tag) against the current main to identify breaking changes. It notes that it does not affect the spec used by other tools, implying it is a safe analysis tool, but it does not explicitly name alternative tools or exclusion conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_endpointsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It makes clear this is a read-only listing operation and specifies the exact output content (method + path). The dependency on a currently loaded spec is also disclosed, which is useful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence with no filler. Every part is informative: what is listed, the format, and the prerequisite state.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter read-only listing tool, the description fully covers purpose, input state, and output shape. No output schema exists, but the description already specifies the returned content as method + path. Nothing essential is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline of 4 applies. The description adds no parameter-specific detail, but none is needed since the schema is empty and complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb (List), a concrete resource (endpoints), and precise output scope (method + path) from the currently loaded API spec. This clearly differentiates it from siblings like call_endpoint, validate_response, and load_api_spec.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It clearly states the tool operates on the currently loaded API spec, which implies it should be used after load_api_spec. It does not explicitly name alternatives or exclusions, but the context is sufficient for an agent to know when this tool is relevant.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_all_testsA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and does it well: it discloses that this makes real calls, that it validates against documented schemas, that some endpoints are skipped, and that includeMutating can create or modify real data on the live API. This is strong behavioral disclosure beyond simple action naming.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no filler. The main action is front-loaded, exclusions follow, and the optional mutating mode is explained last. Every sentence contributes distinct information about scope, alternatives, or side effects.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers scope, exclusions, alternatives, and side effects thoroughly. A minor gap is the lack of stated return format or how results are reported, especially since no output schema exists. Still, the behavior is sufficiently complete for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50%: auth and includeMutating have descriptions, but baseUrl and timeoutMs do not. The description adds valuable semantics for includeMutating (auto-generates params/bodies, risk of real writes) but does not compensate for the undocumented baseUrl and timeoutMs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('calls each one for real and validates the response') on a well-defined resource ('every GET endpoint that needs no required parameters'). It also differentiates itself from the sibling test_endpoint by explicitly saying parameterized endpoints require that other tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: run across all parameter-free GET endpoints, and explicitly routes parameter-dependent endpoints to test_endpoint. It also states exclusions (DELETE/OPTIONS/HEAD never auto-run) and explains when to enable includeMutating, including the default-off warning.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.