Skip to main content
Glama
Automation Coverage MCP

Automation Coverage MCP

by HusneShabbir

Automation Coverage MCP

Coverage-driven test generation for the RHDH plugin forest. This MCP analyzes git changes, reads Istanbul/LCOV numbers, and tells agents which tests to write at which layer. UI work is handed to Playwright MCP as explore-then-write briefs — this server does not drive a browser.

Plugins live at multiple levels (workspaces/<ws>/plugins/<plugin>, backend modules, overlay e2e, RHDH product e2e). Discovery walks that layout instead of hardcoding package names.

What it does

  1. Discover packages across rhdh-plugins, community-plugins, rhdh, rhdh-plugin-export-overlays, and shared Playwright helper repos.

  2. Analyze the current git diff (branch vs origin/main plus uncommitted work) and classify each file (util, React page, backend router, overlay metadata, platform, …). When the branch is clean, pass mode=workspace to scan src instead.

  3. Read coverage (coverage/coverage-final.json or lcov.info) from the cwd / scoped packages only (never a sibling workspace) and intersect uncovered lines with the diff.

  4. Recommend layers using the cheapest-layer-wins ladder (unit → integration → component → plugin Playwright → overlay/cluster). Layers are pluggable YAML.

  5. Emit briefs agents can execute. UI briefs are Playwright MCP prompts: navigate, snapshot, generate locators, verify, then write the spec.

Coverage ranks gaps. It is not a merge gate. Each brief states the failure the test must catch.

Related MCP server: testmcp

Pair with Playwright MCP

Add both servers to Cursor MCP config (~/.cursor/mcp.json, or .cursor/mcp.json in this clone). Playwright needs the testing capability so browser_generate_locator and browser_verify_* are available.

Substitute <ABS_PATH_TO_THIS_CLONE> with the absolute path of this repository on your machine (the directory that contains src/index.ts).

{
  "mcpServers": {
    "automation-coverage": {
      "command": "npx",
      "args": ["tsx", "<ABS_PATH_TO_THIS_CLONE>/src/index.ts"]
    },
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest", "--caps=testing"]
    }
  }
}

If the Cursor workspace is this repository, you can use the relative entry already in .cursor/mcp.json (./src/index.ts) instead of an absolute path.

Agent loop for UI gaps:

generate_playwright_brief → Playwright browser_navigate → browser_snapshot → interact → browser_generate_locator → browser_verify_* → write @playwright/test spec mirroring a neighbor → run until green.

Do not write Playwright from the scenario paragraph alone.

Tools

Tool

Purpose

list_layers

Builtin + YAML layers

discover_packages

Plugin packages at every forest level

analyze_changes

Git diff → package + file kind

get_coverage

Parse Istanbul/LCOV

coverage_gaps

Uncovered ∩ changed lines

inventory_tests

Existing tests (templates to mirror)

recommend_automation

Cheapest layers + Playwright flag

generate_test_plan

Full ordered plan with briefs. mode=workspace fills layers on a clean branch

generate_layer_brief

One layer (unit / integration / …)

generate_playwright_brief

Playwright MCP prompt from UI gaps

Prompts: fill_automation_gaps, playwright_from_coverage, unit_from_coverage.

Pluggable layers

Builtins (from the RHDH test-placement ladder):

id

Ladder

Playwright MCP?

unit

L1 Jest/Vitest

no

integration

L2 startTestBackend

no

component

L3 RTL

no

ui

Plugin-source Playwright

yes

smoke

Overlay native smoke

no

overlay-e2e

Overlay cluster Playwright

yes

cluster-free-e2e

RHDH L4a

yes

cluster-e2e

RHDH L4b

yes

Add or override layers in .automation-coverage.yaml (see config/rhdh-forest.example.yaml):

disabledLayers: [cluster-e2e]
layers:
  - id: contract
    title: HTTP contract
    kind: custom
    cost: s
    fileKinds: [backend-router]
    testGlobs: ["**/*.contract.test.ts"]
    generator: Write a contract test against the public HTTP schema.

Config lookup

  1. AUTOMATION_COVERAGE_CONFIG (absolute path to a YAML file)

  2. .automation-coverage.yaml walking up from cwd

  3. ~/.config/automation-coverage/config.yaml

  4. Auto-detect: walk up from cwd until a parent directory contains sibling clones named rhdh-plugins, community-plugins, rhdh, rhdh-plugin-export-overlays, lightspeed-playwright-e2e, and/or backstage

Forest paths in YAML expand ~ and ${ENV_VAR}. Copy config/rhdh-forest.example.yaml and substitute ${FOREST_ROOT} (or export it) — that value is the parent directory that contains your clones, not rhdh-plugins itself.

Token

Replace with

${FOREST_ROOT}

Absolute path of the folder that contains rhdh-plugins, rhdh, overlays, …

<ABS_PATH_TO_THIS_CLONE>

Absolute path of this automation-coverage-mcp checkout

<plugin-workspace>

Workspace folder name under workspaces/ (e.g. boost)

Omit forest entries you have not cloned. If ${FOREST_ROOT} is left unexpanded, those rows are ignored and auto-detect is used.

Run

cd <ABS_PATH_TO_THIS_CLONE>
npm install
npm test
npm start   # stdio MCP

Produce a coverage report in the plugin workspace before planning (yarn test:all / backstage-cli repo test --coverage). If no report is present, changed lines are treated as uncovered.

Agent workflow

  1. generate_test_plan with cwd set to the plugin workspace. Use mode=workspace when git diff is empty (existing plugin, not a feature branch).

  2. Implement every playwrightMcp: false item by mirroring template

  3. For playwrightMcp: true items, run generate_playwright_brief and drive Playwright MCP

  4. Skip files in skipped (ignored wiring, already covered, non-source, already has a neighbor test)

Available Tools

10 tools
analyze_changesAnalyze git changesB

Parse git diff (branch vs base plus uncommitted) and classify each file into a plugin package and file kind.

Inputs: cwd, base, mode, workspace, +2 more.

coverage_gapsCoverage gaps in changesA

Intersect git-changed lines with uncovered lines from Istanbul/LCOV. Use this before generating tests.

Inputs: cwd, base, mode, workspace, +1 more.

discover_packagesDiscover plugin packagesA

Discover plugin packages at every level of the RHDH forest (workspaces//plugins/, rhdh plugins, overlay e2e, shared Playwright helpers).

Inputs: cwd, workspace.

generate_layer_briefGenerate one layer briefC

Generate a single-layer implementation brief (unit, integration, component, smoke, or a custom YAML layer).

Inputs: layerId (required), cwd, base, mode, +1 more.

generate_playwright_briefGenerate Playwright MCP briefA

Generate the explore-then-write prompt for Playwright MCP from UI coverage gaps. Pair with the playwright MCP server (--caps=testing). Do not write spec code until you have used browser_* tools.

Inputs: cwd, base, mode, workspace.

generate_test_planGenerate full automation planB

Produce a complete, ordered plan: per-gap briefs for unit/integration/component plus Playwright MCP prompts for UI layers. Agents should execute every work item.

Inputs: cwd, base, mode, workspace, +2 more.

get_coverageGet coverage reportB

Parse Istanbul JSON or LCOV coverage from a package or workspace. Returns per-file percents and uncovered line numbers.

Inputs: cwd, reportPath, packagePath.

inventory_testsInventory existing testsA

List existing tests for a package at one layer (or all layers). Used to find a template to mirror.

Inputs: cwd, layerId, packageId, packagePath.

list_layersList test layersA

List pluggable test layers (unit, integration, component, UI/Playwright, smoke, cluster). Custom YAML layers are merged in.

Inputs: cwd.

recommend_automationRecommend test layersB

Given current git changes and coverage numbers, recommend the cheapest test layers and whether Playwright MCP is required.

Inputs: cwd, base, mode, workspace, +1 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 10 tools

    Disambiguation4/5

    Most tools map to distinct pipeline phases: discovery, diff analysis, coverage extraction, gap intersection, inventory, recommendation, and brief generation. The main ambiguity is among generate_test_plan, generate_layer_brief, and generate_playwright_brief, but their descriptions clarify scope.

    Naming Consistency4/5

    Tool names mostly follow a consistent snake_case verb-first pattern such as list_layers, discover_packages, analyze_changes, and generate_test_plan. The exception is coverage_gaps, a noun phrase rather than an imperative verb, which creates a minor inconsistency.

    Tool Count5/5

    With 10 tools, the server is well-scoped for its purpose: discovery, analysis, coverage gap detection, planning, and brief generation. Each tool corresponds to a meaningful step in the workflow, and none feel redundant.

    Completeness5/5

    The tool surface covers the full pipeline from listing layers and discovering packages, through analyzing changes and coverage gaps, to recommending and generating test plans and layer-specific briefs. There are no obvious dead ends or missing operations for the stated purpose.

    Related MCP Connectors

    Related MCP Servers