Server Quality Checklist

Tool Scores

• qa_reportA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the transparency burden. It discloses that this is a read operation, supports filtering, and defaults to the most recent run when runId is omitted. However, it doesn't describe the return format, pagination behavior, or error handling, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short sentences, front-loaded with purpose, then filters, then default behavior. No redundant words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema and annotations, the description adequately communicates the tool's purpose and common usage, but it leaves out details about the response structure and how limit interacts with results, which would be valuable for a tool meant to drill into full details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides descriptions for all six parameters (100% coverage), and the description's mention of filters adds no new meaning beyond the schema. The only additional note about omitting runId is already present in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the verb 'Reads' and identifies the resource as 'a stored run', specifying it provides 'full detail' in contrast to a truncated digest. This clearly distinguishes it from sibling tools that run or discover tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states the use case: 'for drilling into what a run digest truncated', and provides clear filter guidance. It doesn't explicitly mention alternatives, but the contrast with siblings is implicit; this is sufficient context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_run_allA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosure. It discloses the components included, the automatic comparison, the returned digest and diff buckets, and the policy constraint on mutating endpoints. It does not mention whether the run updates the stored baseline or the exact structure of the digest, but overall it provides solid transparency for a complex orchestration tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, using two sentences to convey the core functionality and usage guidance. It avoids fluff, with every phrase earning its place—listing components, mentioning the diff, identifying the blocking bucket, and noting the policy constraint.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (an orchestrator running multiple QA layers) and no output schema, the description adequately explains what the tool does and what it returns (digest plus diff buckets). It highlights the critical 'regressions' bucket but does not detail the full diff bucket structure or potential side effects (e.g., updating the stored run). This is a minor gap, but the description is largely complete for an agent to understand and invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so parameters (role, skipUi, maxPages, services) are fully documented in the schema. The description adds some context by mentioning the components (e.g., UI layers, API part), which aligns with skipUi and services, but it does not add new semantic meaning beyond what the schema already provides. This is the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states what the tool does: a complete QA pass covering discovery, generated API tests, YAML flows, Playwright specs, and frontend crawl, followed by a regression diff against the previous run. This is specific and distinguishes it from the sibling tools (which cover individual components like qa_discover, qa_run_api, etc.).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance, stating 'Use this as the release gate.' It also implies when to use the full run versus the individual sibling tools (e.g., qa_run_api or qa_run_flows), though it doesn't explicitly state when not to use it. The mention of 'regressions' as the blocking bucket further signals its role in release decisions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_diffA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden. It discloses the matching key ('stable test id'), the delta categories, and the default run selection logic, adding behavioral context beyond the schema. It does not mention side effects, but this is a read-only comparison so it is not a major gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action, and every clause adds value. It is compact without being under-specified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (two optional params, no output schema), the description covers the core functionality, output categories, and defaults. It could explicitly describe the return format, but the listed categories imply the response structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description enriches parameter semantics by stating that currentRunId defaults to the latest run and baselineRunId defaults to the most recent earlier run covering the same suites, which the schema does not include.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb 'Compares' and resource 'two stored runs', and goes beyond by explaining the output categories (regressions, fixed, newFailures, etc.), which clearly distinguishes it from sibling tools like qa_run_* and qa_report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by calling regressions 'the release blocker' and specifies default behavior for run selection. It does not explicitly name alternative tools or exclusions, but the context and sibling tool names make the use case clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_run_uiA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses numerous behaviors: missing config handling, failure criteria (navigation error, HTTP >= 400, console errors, failing XHRs, empty render), screenshot on failure, and login failure handling. This is exceptionally transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single well-organized paragraph, front-loaded with 'Two layers against the React frontend.' Each sentence provides essential behavioral details without redundancy, making it appropriately concise for the complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The behavior is thoroughly covered for both modes, including error conditions and outcomes like screenshots. However, it never states what the tool returns upon completion (e.g., a test report or exit codes), which leaves a gap given no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and every parameter has a description, so baseline is 3. The description adds detail for mode values (specs/crawl/both) but does not explain grep, role, maxPages, or startPaths beyond their schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs frontend tests with two modes (specs and crawl), distinguishing it from sibling API test tools like qa_run_api. The title 'Run frontend tests' matches the content, and the description elaborates on the exact scope against the React frontend.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  While it doesn't explicitly name alternatives, it establishes its domain as frontend testing ('against the React frontend') and explains when to use each mode (specs for fixed Playwright specs, crawl for link navigation). It does not mention when not to use it versus qa_run_api or qa_run_all, so some explicit guidance is missing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_run_apiA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility. It discloses deterministic behavior, exact test criteria, 5xx critical failures, mutation-skipping logic, return digest, and artifactsDir storage. This is comprehensive and leaves no major behavioral ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but well-organized, starting with the main action, then test categories, a critical caveat (IMPORTANT), and ending with return/storage info. Every sentence adds unique value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with no output schema, the description is remarkably complete: it covers test types, failure conditions, special policy behavior, return digest, artifact location, and even points to a related tool. It leaves little to no ambiguity about what the tool does and what to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers all three parameters with descriptions (role, services, pathContains). The tool description adds no parameter-specific detail beyond the schema, which already fully documents them, so the baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it generates and executes deterministic API tests from OpenAPI, listing four test categories (contract, authz, robustness, perf). This specific verb+resource+scope distinguishes it from sibling tools like qa_run_ui and qa_run_flows.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool (for API testing) and explicitly directs the agent to call qa_discover to see skipped mutations under a readOnly policy. It does not explicitly contrast with qa_run_all, but it provides strong contextual guidance on the workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_run_flowsA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden. It discloses sequential execution, state sharing, the ability to chain calls with interpolation/capture/expectations/waitMs, the fact that destructive operations are allowed only here, and the failure result format (failing step name, trace, captured variables).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences packed with essential information, no fluff. Front-loaded with the main action, then covers semantics, safety, and result format efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description explains the result format on failure and implies success results. It covers execution mode, state sharing, destructive operation safety, and flow semantics. Adequate for a complex tool with no annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are already documented in the schema. The description adds no extra parameter details, but it doesn't need to. Baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Executes the hand-written business scenarios in tests/flows/*.yaml sequentially', which is a specific verb+resource. It also distinguishes itself from siblings by noting it is 'the only place where destructive operations are allowed' and emphasizes the cross-service flow chaining.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides strong context: flows share state and must never be run in parallel, and this is the only tool where destructive operations are allowed. It implies when to use it over alternatives but does not explicitly name sibling tools or state when not to use it beyond the parallel warning.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_discoverA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden. It discloses that it fetches OpenAPI/Swagger documents, never sends requests to the endpoint APIs themselves, reports on services that could not be reached, and explains safety-policy skips. This is a comprehensive behavioral profile.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core action and ending with usage advice. It includes only relevant details and no filler, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description adequately summarizes the return values (endpoint count, assertion counts, skipped operations, unreachable services). It also includes safety behavior and usage context, making the description self-sufficient for an agent to understand the tool's outcomes.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already fully documents the single optional 'services' parameter with its description. The tool description adds no additional information about the parameter beyond the schema, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies this as a discovery tool for the OpenAPI/Swagger documents of configured services. It specifies the verb 'fetches' and the resource, and reports a specific set of metrics (endpoint count, assertions, skipped operations, unreachable services). This distinguishes it from sibling tools that run tests or generate reports.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to 'Use this first to check coverage before running anything', which tells the agent when to invoke this tool relative to the test-running siblings. It also notes that it 'runs no test and sends no request to the endpoints themselves', clarifying that it is safe for pre-flight checks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.