Automation Coverage MCP
Supports the Backstage ecosystem (including Red Hat Developer Hub plugin development) by discovering plugin packages, analyzing git changes and coverage reports, and recommending/generating test plans and Playwright briefs across unit, integration, component, and e2e layers.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Automation Coverage MCP Generate a test plan for my current changes"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Automation Coverage MCP
Coverage-driven test generation for the RHDH plugin forest. This MCP analyzes git changes, reads Istanbul/LCOV numbers, and tells agents which tests to write at which layer. UI work is handed to Playwright MCP as explore-then-write briefs — this server does not drive a browser.
Plugins live at multiple levels (workspaces/<ws>/plugins/<plugin>, backend modules, overlay e2e, RHDH product e2e). Discovery walks that layout instead of hardcoding package names.
What it does
Discover packages across
rhdh-plugins,community-plugins,rhdh,rhdh-plugin-export-overlays, and shared Playwright helper repos.Analyze the current git diff (branch vs
origin/mainplus uncommitted work) and classify each file (util, React page, backend router, overlay metadata, platform, …). When the branch is clean, passmode=workspaceto scansrcinstead.Read coverage (
coverage/coverage-final.jsonorlcov.info) from the cwd / scoped packages only (never a sibling workspace) and intersect uncovered lines with the diff.Recommend layers using the cheapest-layer-wins ladder (unit → integration → component → plugin Playwright → overlay/cluster). Layers are pluggable YAML.
Emit briefs agents can execute. UI briefs are Playwright MCP prompts: navigate, snapshot, generate locators, verify, then write the spec.
Coverage ranks gaps. It is not a merge gate. Each brief states the failure the test must catch.
Related MCP server: testmcp
Pair with Playwright MCP
Add both servers to Cursor MCP config (~/.cursor/mcp.json, or .cursor/mcp.json in this clone). Playwright needs the testing capability so browser_generate_locator and browser_verify_* are available.
Substitute <ABS_PATH_TO_THIS_CLONE> with the absolute path of this repository on your machine (the directory that contains src/index.ts).
{
"mcpServers": {
"automation-coverage": {
"command": "npx",
"args": ["tsx", "<ABS_PATH_TO_THIS_CLONE>/src/index.ts"]
},
"playwright": {
"command": "npx",
"args": ["@playwright/mcp@latest", "--caps=testing"]
}
}
}If the Cursor workspace is this repository, you can use the relative entry already in .cursor/mcp.json (./src/index.ts) instead of an absolute path.
Agent loop for UI gaps:
generate_playwright_brief → Playwright browser_navigate → browser_snapshot → interact → browser_generate_locator → browser_verify_* → write @playwright/test spec mirroring a neighbor → run until green.
Do not write Playwright from the scenario paragraph alone.
Tools
Tool | Purpose |
| Builtin + YAML layers |
| Plugin packages at every forest level |
| Git diff → package + file kind |
| Parse Istanbul/LCOV |
| Uncovered ∩ changed lines |
| Existing tests (templates to mirror) |
| Cheapest layers + Playwright flag |
| Full ordered plan with briefs. |
| One layer (unit / integration / …) |
| Playwright MCP prompt from UI gaps |
Prompts: fill_automation_gaps, playwright_from_coverage, unit_from_coverage.
Pluggable layers
Builtins (from the RHDH test-placement ladder):
id | Ladder | Playwright MCP? |
| L1 Jest/Vitest | no |
| L2 | no |
| L3 RTL | no |
| Plugin-source Playwright | yes |
| Overlay native smoke | no |
| Overlay cluster Playwright | yes |
| RHDH L4a | yes |
| RHDH L4b | yes |
Add or override layers in .automation-coverage.yaml (see config/rhdh-forest.example.yaml):
disabledLayers: [cluster-e2e]
layers:
- id: contract
title: HTTP contract
kind: custom
cost: s
fileKinds: [backend-router]
testGlobs: ["**/*.contract.test.ts"]
generator: Write a contract test against the public HTTP schema.Config lookup
AUTOMATION_COVERAGE_CONFIG(absolute path to a YAML file).automation-coverage.yamlwalking up fromcwd~/.config/automation-coverage/config.yamlAuto-detect: walk up from
cwduntil a parent directory contains sibling clones namedrhdh-plugins,community-plugins,rhdh,rhdh-plugin-export-overlays,lightspeed-playwright-e2e, and/orbackstage
Forest paths in YAML expand ~ and ${ENV_VAR}. Copy config/rhdh-forest.example.yaml and substitute ${FOREST_ROOT} (or export it) — that value is the parent directory that contains your clones, not rhdh-plugins itself.
Token | Replace with |
| Absolute path of the folder that contains |
| Absolute path of this |
| Workspace folder name under |
Omit forest entries you have not cloned. If ${FOREST_ROOT} is left unexpanded, those rows are ignored and auto-detect is used.
Run
cd <ABS_PATH_TO_THIS_CLONE>
npm install
npm test
npm start # stdio MCPProduce a coverage report in the plugin workspace before planning (yarn test:all / backstage-cli repo test --coverage). If no report is present, changed lines are treated as uncovered.
Agent workflow
generate_test_planwithcwdset to the plugin workspace. Usemode=workspacewhen git diff is empty (existing plugin, not a feature branch).Implement every
playwrightMcp: falseitem by mirroringtemplateFor
playwrightMcp: trueitems, rungenerate_playwright_briefand drive Playwright MCPSkip files in
skipped(ignored wiring, already covered, non-source, already has a neighbor test)
Available Tools
10 toolsanalyze_changesAnalyze git changesB
Parse git diff (branch vs base plus uncommitted) and classify each file into a plugin package and file kind.
Inputs: cwd, base, mode, workspace, +2 more.
coverage_gapsCoverage gaps in changesA
Intersect git-changed lines with uncovered lines from Istanbul/LCOV. Use this before generating tests.
Inputs: cwd, base, mode, workspace, +1 more.
discover_packagesDiscover plugin packagesA
Discover plugin packages at every level of the RHDH forest (workspaces//plugins/, rhdh plugins, overlay e2e, shared Playwright helpers).
Inputs: cwd, workspace.
generate_layer_briefGenerate one layer briefC
Generate a single-layer implementation brief (unit, integration, component, smoke, or a custom YAML layer).
Inputs: layerId (required), cwd, base, mode, +1 more.
generate_playwright_briefGenerate Playwright MCP briefA
Generate the explore-then-write prompt for Playwright MCP from UI coverage gaps. Pair with the playwright MCP server (--caps=testing). Do not write spec code until you have used browser_* tools.
Inputs: cwd, base, mode, workspace.
generate_test_planGenerate full automation planB
Produce a complete, ordered plan: per-gap briefs for unit/integration/component plus Playwright MCP prompts for UI layers. Agents should execute every work item.
Inputs: cwd, base, mode, workspace, +2 more.
get_coverageGet coverage reportB
Parse Istanbul JSON or LCOV coverage from a package or workspace. Returns per-file percents and uncovered line numbers.
Inputs: cwd, reportPath, packagePath.
inventory_testsInventory existing testsA
List existing tests for a package at one layer (or all layers). Used to find a template to mirror.
Inputs: cwd, layerId, packageId, packagePath.
list_layersList test layersA
List pluggable test layers (unit, integration, component, UI/Playwright, smoke, cluster). Custom YAML layers are merged in.
Inputs: cwd.
recommend_automationRecommend test layersB
Given current git changes and coverage numbers, recommend the cheapest test layers and whether Playwright MCP is required.
Inputs: cwd, base, mode, workspace, +1 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 10 tools
Most tools map to distinct pipeline phases: discovery, diff analysis, coverage extraction, gap intersection, inventory, recommendation, and brief generation. The main ambiguity is among generate_test_plan, generate_layer_brief, and generate_playwright_brief, but their descriptions clarify scope.
Tool names mostly follow a consistent snake_case verb-first pattern such as list_layers, discover_packages, analyze_changes, and generate_test_plan. The exception is coverage_gaps, a noun phrase rather than an imperative verb, which creates a minor inconsistency.
With 10 tools, the server is well-scoped for its purpose: discovery, analysis, coverage gap detection, planning, and brief generation. Each tool corresponds to a meaningful step in the workflow, and none feel redundant.
The tool surface covers the full pipeline from listing layers and discovering packages, through analyzing changes and coverage gaps, to recommending and generating test plans and layer-specific briefs. There are no obvious dead ends or missing operations for the stated purpose.
Related MCP Connectors
Generates unit tests for Python code with coverage before/after reports and concrete edge cases.
Approved test intent, reviewed Playwright automation and run evidence, inside your editor.
Test analytics for AI agents: test history, flaky tests, failure clusters, coverage.
QA platform for agents: coverage signals, in-repo test plans, verified tests and release governance.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceAutomates repository analysis, test planning, and generation of end-to-end tests with Playwright, acting as an intelligent quality assistant.18 npmMIT
- AlicenseAqualityDmaintenanceWraps existing test frameworks (Jest, Vitest, Pytest) and exposes structured, LLM-optimized results via MCP tools with progressive disclosure and diff-aware execution.111MIT
- AlicenseAqualityBmaintenanceTest coverage intelligence MCP server for TypeScript and JavaScript projects. Analyzes coverage reports to find untested functions, computes cyclomatic complexity, and suggests test cases locally.457 npm3MIT
- AlicenseAqualityDmaintenanceEnables AI assistants to summarize Playwright test runs, identify flaky tests, heuristically triage failures, generate Playwright test stubs, and retrieve recent GitHub CI status through MCP tool calls.5MIT