Supports the Backstage ecosystem (including Red Hat Developer Hub plugin development) by discovering plugin packages, analyzing git changes and coverage reports, and recommending/generating test plans and Playwright briefs across unit, integration, component, and e2e layers.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Automation Coverage MCP Generate a test plan for my current changes"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Coverage-driven test generation for the RHDH plugin forest. This MCP analyzes git changes, reads Istanbul/LCOV numbers, and tells agents which tests to write at which layer. UI work is handed to Playwright MCP as explore-then-write briefs — this server does not drive a browser.
Plugins live at multiple levels (workspaces/<ws>/plugins/<plugin>, backend modules, overlay e2e, RHDH product e2e). Discovery walks that layout instead of hardcoding package names.
Discover packages across rhdh-plugins, community-plugins, rhdh, rhdh-plugin-export-overlays, and shared Playwright helper repos.
Analyze the current git diff (branch vs origin/main plus uncommitted work) and classify each file (util, React page, backend router, overlay metadata, platform, …). When the branch is clean, pass mode=workspace to scan src instead.
Read coverage (coverage/coverage-final.json or lcov.info) from the cwd / scoped packages only (never a sibling workspace) and intersect uncovered lines with the diff.
Recommend layers using the cheapest-layer-wins ladder (unit → integration → component → plugin Playwright → overlay/cluster). Layers are pluggable YAML.
Emit briefs agents can execute. UI briefs are Playwright MCP prompts: navigate, snapshot, generate locators, verify, then write the spec.
Coverage ranks gaps. It is not a merge gate. Each brief states the failure the test must catch.
Related MCP server: Quality MCP
Add both servers to Cursor MCP config (~/.cursor/mcp.json, or .cursor/mcp.json in this clone). Playwright needs the testing capability so browser_generate_locator and browser_verify_* are available.
Substitute <ABS_PATH_TO_THIS_CLONE> with the absolute path of this repository on your machine (the directory that contains src/index.ts).
{
"mcpServers": {
"automation-coverage": {
"command": "npx",
"args": ["tsx", "<ABS_PATH_TO_THIS_CLONE>/src/index.ts"]
},
"playwright": {
"command": "npx",
"args": ["@playwright/mcp@latest", "--caps=testing"]
}
}
}If the Cursor workspace is this repository, you can use the relative entry already in .cursor/mcp.json (./src/index.ts) instead of an absolute path.
Agent loop for UI gaps:
generate_playwright_brief → Playwright browser_navigate → browser_snapshot → interact → browser_generate_locator → browser_verify_* → write @playwright/test spec mirroring a neighbor → run until green.
Do not write Playwright from the scenario paragraph alone.
Tool | Purpose |
| Builtin + YAML layers |
| Plugin packages at every forest level |
| Git diff → package + file kind |
| Parse Istanbul/LCOV |
| Uncovered ∩ changed lines |
| Existing tests (templates to mirror) |
| Cheapest layers + Playwright flag |
| Full ordered plan with briefs. |
| One layer (unit / integration / …) |
| Playwright MCP prompt from UI gaps |
Prompts: fill_automation_gaps, playwright_from_coverage, unit_from_coverage.
Builtins (from the RHDH test-placement ladder):
id | Ladder | Playwright MCP? |
| L1 Jest/Vitest | no |
| L2 | no |
| L3 RTL | no |
| Plugin-source Playwright | yes |
| Overlay native smoke | no |
| Overlay cluster Playwright | yes |
| RHDH L4a | yes |
| RHDH L4b | yes |
Add or override layers in .automation-coverage.yaml (see config/rhdh-forest.example.yaml):
disabledLayers: [cluster-e2e]
layers:
- id: contract
title: HTTP contract
kind: custom
cost: s
fileKinds: [backend-router]
testGlobs: ["**/*.contract.test.ts"]
generator: Write a contract test against the public HTTP schema.AUTOMATION_COVERAGE_CONFIG (absolute path to a YAML file)
.automation-coverage.yaml walking up from cwd
~/.config/automation-coverage/config.yaml
Auto-detect: walk up from cwd until a parent directory contains sibling clones named rhdh-plugins, community-plugins, rhdh, rhdh-plugin-export-overlays, lightspeed-playwright-e2e, and/or backstage
Forest paths in YAML expand ~ and ${ENV_VAR}. Copy config/rhdh-forest.example.yaml and substitute ${FOREST_ROOT} (or export it) — that value is the parent directory that contains your clones, not rhdh-plugins itself.
Token | Replace with |
| Absolute path of the folder that contains |
| Absolute path of this |
| Workspace folder name under |
Omit forest entries you have not cloned. If ${FOREST_ROOT} is left unexpanded, those rows are ignored and auto-detect is used.
cd <ABS_PATH_TO_THIS_CLONE>
npm install
npm test
npm start # stdio MCPProduce a coverage report in the plugin workspace before planning (yarn test:all / backstage-cli repo test --coverage). If no report is present, changed lines are treated as uncovered.
generate_test_plan with cwd set to the plugin workspace. Use mode=workspace when git diff is empty (existing plugin, not a feature branch).
Implement every playwrightMcp: false item by mirroring template
For playwrightMcp: true items, run generate_playwright_brief and drive Playwright MCP
Skip files in skipped (ignored wiring, already covered, non-source, already has a neighbor test)
Parse git diff (branch vs base plus uncommitted) and classify each file into a plugin package and file kind.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| base | No | Base ref, default origin/main | |
| mode | No | diff = git changes only; workspace = scan source when the branch is clean | |
| workspace | No | Workspace name filter, e.g. boost | |
| includeUntracked | No | ||
| includeUncommitted | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses the key scoping behavior — branch vs base plus uncommitted — and the classification outcome, which implies a read-only analysis. However, it does not mention side effects, failure behavior, or whether any filesystem or repository state is modified.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One tight sentence with the main verb and resource front-loaded. The parenthetical scope adds essential detail without wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This tool has 6 parameters, no output schema, no annotations, and multiple siblings, but the description gives only a high-level behavior. It does not explain the mode parameter's workspace scanning behavior, default base handling, prerequisites like a git repository, or what the classification result looks like.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 50%, and the description partially compensates by explaining the diff source ('branch vs base' maps to base, 'uncommitted' maps to includeUncommitted). It does not clarify cwd, includeUntracked, or mode beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: parsing a git diff and classifying files into a plugin package and file kind. This clearly differentiates the tool from siblings like list_layers, get_coverage, and generate_test_plan, which perform different operations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no explicit guidance on when to use this tool versus its siblings, and no conditions or prerequisites are stated. It implies usage from the verb 'parse', but does not say when this is the right choice or when another tool like discover_packages should be preferred.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Intersect git-changed lines with uncovered lines from Istanbul/LCOV. Use this before generating tests.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| base | No | ||
| mode | No | ||
| workspace | No | ||
| reportPath | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of behavioral disclosure, yet it only names the intersection logic. It does not clarify whether the tool requires a git repo, reads files from disk, mutates anything, or what it returns in edge cases.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler; the core operation is front-loaded and the usage context follows. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-parameter tool with no annotations and no output schema, this definition is too sparse. An agent can guess the intent but not confidently configure cwd, base, mode, or workspace, nor anticipate failure or output shape.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must explain the five parameters. It indirectly implies reportPath refers to LCOV and base/workspace relate to git changes, but mode, cwd, and workspace semantics are left entirely uninferred.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a concrete operation — intersecting git-changed lines with uncovered lines from Istanbul/LCOV — and a clear resource. This distinguishes it from siblings like get_coverage or analyze_changes, which address coverage or change analysis separately.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Use this before generating tests" explicitly states when to invoke the tool. It does not mention alternatives or exclusions, but the timing guidance is unambiguous enough to route an agent correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Discover plugin packages at every level of the RHDH forest (workspaces//plugins/, rhdh plugins, overlay e2e, shared Playwright helpers).
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | Starting directory for forest detection | |
| workspace | No | Only packages in this workspace, e.g. boost |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It does convey that discovery spans multiple forest locations, but it does not state what the tool returns, whether it is read-only, or how cwd/workspace affect traversal. This leaves meaningful behavioral gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence conveys the purpose and scope, with a parenthetical of concrete path patterns that earns its place. There is no filler or repetition of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is reasonably complete for a simple two-optional-parameter discovery tool, but there is no output schema and no mention of the return shape or how results are organized. An agent would still be uncertain about what the tool actually yields.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents cwd and workspace clearly. The description adds helpful forest-scope context but does not materially clarify parameter semantics beyond what the schema provides, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Discover') and a concrete resource ('plugin packages'), and defines the exact scope with the RHDH forest levels and examples. This distinguishes it clearly from sibling tools like list_layers or inventory_tests, which address different artifacts.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is for enumerating plugin packages across all defined RHDH forest levels, but it does not explicitly state when to use this tool versus alternatives or mention any exclusions. Context is present, but selection guidance is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate a single-layer implementation brief (unit, integration, component, smoke, or a custom YAML layer).
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| base | No | ||
| mode | No | ||
| layerId | Yes | Layer id from list_layers | |
| workspace | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits itself. It only states the action without mentioning side effects (e.g., file writes), required permissions, or output characteristics. The parameters cwd and workspace hint at file-system interaction, but no behavioral details are given.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no filler, front-loading the core purpose and enumerating valid layer types. It is concise and well-structured, though slightly too brief to convey all necessary context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 5 parameters, including one enum and no output schema, the description is too thin. It omits parameter semantics, usage conditions, and return expectations, so an agent would lack crucial information for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 20% (only layerId has a schema description). The tool description does not explain any parameter beyond the schema's layerId note, so the 4 other parameters (cwd, base, mode, workspace) remain completely undocumented. The description fails to compensate for the low coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool generates a single-layer implementation brief and lists the possible layer types (unit, integration, component, smoke, custom YAML). The verb 'generate' and resource 'single-layer implementation brief' are specific, but it does not explicitly differentiate from sibling tools like generate_test_plan or generate_playwright_brief, so it misses the top score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus its siblings. It does not mention alternatives, prerequisites, or scenario-specific advice, leaving the agent to infer usage from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate the explore-then-write prompt for Playwright MCP from UI coverage gaps. Pair with the playwright MCP server (--caps=testing). Do not write spec code until you have used browser_* tools.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| base | No | ||
| mode | No | ||
| workspace | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations to rely on, the description provides useful behavioral context: this tool generates an explore-then-write prompt rather than spec code, requires pairing with the Playwright MCP server, and imposes an ordering constraint. It does not detail return format or side effects, but the generation role is reasonably well conveyed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with no unnecessary filler. Key information—what is generated, from what input, and the usage constraint—is front-loaded and compact.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description leaves major invocation details unspecified: parameter semantics, expected output shape, and any behavior differences between modes. Given no annotations and no output schema, the definition is not complete enough for reliable correct usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description does not explain cwd, base, mode, or workspace. It only loosely implies that coverage gaps feed the prompt, which is insufficient for an agent to understand distinct parameter meanings or how diff vs workspace mode affects behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and object: 'Generate the explore-then-write prompt for Playwright MCP from UI coverage gaps.' It distinguishes this from sibling tools like generate_test_plan and generate_layer_brief by naming Playwright MCP and the 'explore-then-write' prompt format.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly indicates when to use the tool: when UI coverage gaps exist and a Playwright MCP brief is needed. It also gives operational guidance by specifying the server pairing and sequencing ('Do not write spec code until you have used browser_* tools'), though it does not explicitly contrast it with sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Produce a complete, ordered plan: per-gap briefs for unit/integration/component plus Playwright MCP prompts for UI layers. Agents should execute every work item.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| base | No | ||
| mode | No | workspace scans src when git diff is empty (or always when set) | |
| workspace | No | ||
| reportPath | No | ||
| includeUntracked | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden and it does clarify that the tool produces a plan and that agents should execute work items, implying no execution side effects. However, it is silent on file writes via reportPath, required permissions, and the exact shape of the output, so behavioral disclosure is only partial.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is short, front-loaded with the deliverable, and contains no filler. The second sentence is an operational directive rather than tool documentation, but it is compact and non-redundant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a six-parameter tool with no annotations, no output schema, and many related siblings, this description is incomplete: it does not explain where the plan is written, how parameters control scope, or how this tool relates to generate_layer_brief and generate_playwright_brief. It leaves too much for the agent to infer.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 17% and the description does not compensate: cwd, base, workspace, reportPath, and includeUntracked are never given meaning. Only mode has a schema description, so an agent has almost no information to set these parameters confidently.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Produce') and a concrete deliverable ('complete, ordered plan') with enumerated contents (per-gap briefs for unit/integration/component plus Playwright MCP prompts). This makes it easy to distinguish from sibling brief generators, which each target a single layer.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'complete, ordered plan' implies this is the full-plan tool, but no explicit when-to-use or when-not-to-use guidance is given. An agent must infer the relationship to generate_layer_brief and generate_playwright_brief rather than being told when to choose the aggregated plan over individual briefs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Parse Istanbul JSON or LCOV coverage from a package or workspace. Returns per-file percents and uncovered line numbers.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| reportPath | No | Explicit coverage-final.json or lcov.info path | |
| packagePath | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It does disclose the non-mutating nature via 'Parse' and specifies the return content. However, it does not mention whether it searches for reports automatically, how it behaves with missing files, or whether any workspace state is read. This adds some value but leaves significant behavioral gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler. The first sentence states the primary action and input types; the second states the output. Information is front-loaded and every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Without annotations, an output schema, or parameter descriptions for two of three fields, the description leaves important gaps: how to choose between cwd/packagePath/reportPath, default behavior, and the exact shape of the per-file report. For a tool with multiple optional parameters, this is not complete enough for reliable invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 33%: only 'reportPath' has a description. The tool description adds context for 'package or workspace' and clarifies 'reportPath' as an explicit path, but it does not explain how 'cwd' and 'packagePath' interact or when they should be supplied. The description only partially compensates for the low schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies a specific verb ('Parse'), resource ('Istanbul JSON or LCOV coverage'), and result ('per-file percents and uncovered line numbers'). It is unambiguous about what the tool does, though it does not explicitly distinguish itself from the sibling tool 'coverage_gaps'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no guidance on when to use this tool versus alternatives such as 'coverage_gaps', 'analyze_changes', or 'inventory_tests'. It implies usage for parsing coverage files but does not state prerequisites, fallback scenarios, or when a sibling would be more appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List existing tests for a package at one layer (or all layers). Used to find a template to mirror.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| layerId | No | ||
| packageId | No | Package id from discover_packages, e.g. plugins:workspaces/scorecard/plugins/scorecard | |
| packagePath | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the behavioral burden. The verb 'List' communicates a read-only operation, and 'Used to find a template to mirror' hints that the result can serve as a template. However, the description does not disclose the output shape, what matching/scanning behavior is involved, or any side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences with no filler. The core action is front-loaded and the purpose adds useful intent without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With four optional parameters, no required parameters, no annotations, and no output schema, the description needs to explain parameter selection and return value. It clarifies layer/package concepts and the template use case, but it does not explain how to choose between cwd, packageId, and packagePath, nor what the returned inventory looks like.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 25%, with cwd, layerId, and packagePath left undocumented. The description compensates partially by mentioning 'one layer (or all layers)' for layerId and the schema example clarifies packageId, but cwd and packagePath semantics remain unexplained in both the schema and description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'List existing tests for a package at one layer (or all layers)'. It also gives an explicit use case, 'Used to find a template to mirror', which distinguishes this inventory tool from sibling analysis/generation tools like get_coverage or generate_test_plan.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Used to find a template to mirror' provides a clear scenario for when this tool is appropriate. It does not explicitly name alternatives or state when not to use it, but the context is sufficient for an agent to see this is the inventory step rather than an analysis or generation step.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List pluggable test layers (unit, integration, component, UI/Playwright, smoke, cluster). Custom YAML layers are merged in.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | Directory used to load automation-coverage.yaml |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral burden. It does disclose that custom YAML layers are merged in, which is useful, but it does not describe return format, pagination, error behavior, or whether any side effects occur. For a simple list operation, this is adequate but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences with no filler. The core action and layer types are front-loaded, and the custom YAML merge behavior is an important extra detail that earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with one optional parameter and no output schema, the description is nearly complete. It names the resource, enumerates the layer types, and notes the YAML merge behavior. It lacks only explicit guidance on when to choose this over sibling tools, which is a minor gap for such a simple listing operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single parameter cwd already has a complete description in the input schema ('Directory used to load automation-coverage.yaml'), so schema coverage is 100%. The tool description adds no additional parameter meaning but does not need to, given the schema is sufficient.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('List') and resource ('pluggable test layers'), enumerating the exact layer types: unit, integration, component, UI/Playwright, smoke, and cluster. It also notes that custom YAML layers are merged, which distinguishes it from simpler listing tools and siblings like get_coverage or inventory_tests.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance is given on when to use this tool versus alternatives such as inventory_tests or generate_layer_brief. The context is clear that it lists layers, but there are no exclusions, prerequisites, or sibling comparisons to help an agent select it confidently.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Given current git changes and coverage numbers, recommend the cheapest test layers and whether Playwright MCP is required.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | ||
| base | No | ||
| mode | No | ||
| workspace | No | ||
| reportPath | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It clarifies that the tool analyzes existing git changes and coverage data and returns a recommendation, implying a read-only analysis. However, it does not state side effects, permissions, or how data is sourced.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence with zero wasted words. It front-loads the input condition and then states both outputs. Every part of the sentence adds meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 5 parameters, no output schema, and no annotations, the description is under-specified. It lacks parameter details, expected output shape, and differentiation from sibling tools, making it hard for an agent to invoke correctly without external knowledge.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description only gestures at two conceptual inputs ('git changes' and 'coverage numbers') without mapping them to specific parameters like cwd, base, mode, workspace, or reportPath. It fails to explain the mode enum or the optionality/role of each parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action: 'recommend the cheapest test layers and whether Playwright MCP is required,' with clear input context ('current git changes and coverage numbers'). It is distinct from siblings like generate_test_plan or generate_layer_brief, though it does not name them explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives an implied usage condition (when you have git changes and coverage numbers), but it does not mention alternatives or provide when-not-to-use guidance. With overlapping siblings, this leaves room for ambiguity.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0analyze_changescoverage_gapsdiscover_packagesgenerate_layer_briefgenerate_playwright_briefgenerate_test_planget_coverageinventory_testslist_layersrecommend_automationScored across 10 tools
Most tools map to distinct pipeline phases: discovery, diff analysis, coverage extraction, gap intersection, inventory, recommendation, and brief generation. The main ambiguity is among generate_test_plan, generate_layer_brief, and generate_playwright_brief, but their descriptions clarify scope.
Tool names mostly follow a consistent snake_case verb-first pattern such as list_layers, discover_packages, analyze_changes, and generate_test_plan. The exception is coverage_gaps, a noun phrase rather than an imperative verb, which creates a minor inconsistency.
With 10 tools, the server is well-scoped for its purpose: discovery, analysis, coverage gap detection, planning, and brief generation. Each tool corresponds to a meaningful step in the workflow, and none feel redundant.
The tool surface covers the full pipeline from listing layers and discovering packages, through analyzing changes and coverage gaps, to recommending and generating test plans and layer-specific briefs. There are no obvious dead ends or missing operations for the stated purpose.
Generates unit tests for Python code with coverage before/after reports and concrete edge cases.
Approved test intent, reviewed Playwright automation and run evidence, inside your editor.
Writes adversarial test suites for AI-built code. Your agent's test engineer.
Direct access to Cypress tests results and accessibility reports in your AI workflow.
