Server Quality Checklist

Tool Scores

• run_testsA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. The single sentence only states the action and output, without disclosing side effects, defaults, backgrounding behavior, or permissions. It says results are structured but gives no further behavioral detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no unnecessary words, immediately stating the action and output. It is appropriately concise and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 12 parameters and no output schema, yet the description does not explain return values or the overall workflow (e.g., how background runs interact with get_run_status). It relies heavily on parameter descriptions, leaving the agent to piece together usage context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 92%, so baseline is 3 even without parameter info in the description. The main description adds no parameter semantics beyond the schema's detailed per-parameter explanations.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb-resource pair ('Run Playwright tests') and clearly states the output ('return structured results'). This distinguishes it from sibling tools like get_run_status or list_tests, which handle monitoring or listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The main description implies the primary use case (running tests). The schema's 'wait' parameter explicitly mentions using get_run_status for background runs, providing an alternative. However, there are no explicit exclusions or when-not-to-use instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_failed_testsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It conveys that this is a read operation returning error messages and attachment paths, but it does not disclose behavior when no previous run exists, whether results are ordered/filtered, or any side effects. It adds some value but remains minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single front-loaded sentence that states the action, the target, and the key output fields. Every word contributes value, and there is no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter getter with no output schema, the description provides enough information to select and invoke the tool, including what the result contains. It only lacks explicit edge-case behavior such as the absence of a prior run, but this does not make the tool unusable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The sole parameter workingDirectory has 100% schema description coverage, so the baseline applies. The tool description does not add any parameter-specific meaning, but the schema already documents the parameter thoroughly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Return') and a clear resource ('failed tests from the last run'), and it names the included contents ('error messages and attachment paths'). This distinguishes it from sibling tools like get_run_status and get_test_attachment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'From the last run' implies usage after test execution, giving some context, but the description does not explicitly state when to use this tool over siblings such as get_run_status or when not to use it. The guidance is inferred rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_test_attachmentA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It indicates a read-only operation by using 'Read', but does not mention potential errors (e.g., missing test attachment), format of returned content, or any limitations. This is adequate but leaves room for more transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that conveys the essential purpose without redundancy. It is front-loaded with the verb and resource, and every word contributes to understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read operation with no output schema and no annotations, the description provides enough to understand the tool's primary function. However, it lacks detail on return format or failure behavior, which could be important for agent usage. Given the low complexity, a 4 is appropriate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers all parameters with descriptions, so baseline 3 applies. The tool description itself adds little beyond the schema, though it reinforces the relationship between testTitle and attachmentName. No additional semantics are provided beyond what the schema already states.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the verb ('Read'), the resource ('content of a named attachment'), and the scope ('for a specific test from the last run'). It distinguishes the tool from siblings like run_tests and list_tests, which focus on execution and listing, not reading attachments.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (to read an attachment for a specific test from the last run) and provides clear context. It does not explicitly list alternatives or exclusions, but the sibling tools are sufficiently distinct that no confusion arises.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_testsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the transparency burden. It discloses the key behavioral trait of not running tests and mentions the output (spec file, tags). However, it does not elaborate on side effects, scope limits, or return format, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that is front-loaded with the verb and resource, with zero filler. It efficiently communicates the core purpose and key distinction.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with only two optional parameters and no output schema. The description covers the purpose, output content, and the non-execution behavior, which is sufficient for basic use. It could mention the workingDirectory scope more explicitly, but the schema handles that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the parameters are well-documented in the schema. The description adds no additional parameter semantics but does mention what output fields are included, which is not directly in the schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('list') and resource ('all tests') and clarifies the output ('spec file and tags'). The phrase 'without running them' explicitly differentiates it from the sibling tool 'run_tests'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by stating 'without running them,' which signals this is for inspection only and contrasts with run_tests. It lacks explicit 'use this instead of X' instructions but provides clear context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_run_statusA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses a key behavioral limitation ('does not inspect unrelated OS processes') and explains the fallback behavior when no tracked run exists ('returns idle with current results.json metadata'). It doesn't enumerate possible status values or confirm read-only nature, but the negative statement and fallback provide meaningful transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, starts with the core purpose, and efficiently conveys the parameter alternatives and edge-case behavior in the second sentence. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple status-check tool with no output schema, the description covers the main invocation scenarios, the fallback, and explicitly scopes what it does not do. It could specify possible status values, but the mention of results.json metadata gives enough context for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides 100% coverage with detailed descriptions for both runId and workingDirectory, including their relationship. The description paraphrases the runId/workingDirectory conditional logic but adds no new parameter-level meaning beyond the schema. Therefore, baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Return status for a tracked Playwright run.' It uses a specific verb and resource, and distinguishes itself from generic process inspection by noting it only handles tracked runs, not unrelated OS processes. This separates it from siblings like run_tests and get_failed_tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit conditional guidance: pass runId for a specific background run, or omit it to use workingDirectory for the latest tracked run. It also explains the edge case when no tracked run exists. However, it does not explicitly name alternative sibling tools or say when not to use this tool, so it's clear but not fully comprehensive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.