qa-metrics-mcp

Ask your AI assistant about your test suite in plain English, and get answers straight from your CI test reports.

qa-metrics-mcp is a Model Context Protocol server that turns JUnit XML test reports into QA metrics Claude, Cursor, or any MCP client can query:

*"What broke in last night's run, and which commit caused it?"* *"Which tests are flaky, and how should we fix them?"* *"Write the nightly QA summary for Slack."* *"Build me a dashboard for the last two weeks."*

No dashboards to log into, no spreadsheets to update by hand. The assistant calls the tools, reads the numbers, and explains them. ##Demo https://github.com/user-attachments/assets/6e7a83ce-0a69-40ab-8ac8-efbc7a57034e 

Why this exists

Every QA team I've led spent hours a week on the same work: scrolling CI logs to tell a real regression from a flaky test, figuring out which build introduced a failure, and writing the same status update every morning. This server automates that triage so engineers spend their time fixing issues, not finding them.

Related MCP server: Tesults MCP

What it does

Plus a nightly\_triage prompt that chains the tools into a full morning triage: executive summary, regressions with draft bug titles, flakes to quarantine, and incidents for DevOps.

The smart parts

• Outages don't pollute flake scores. When one error signature hits 3+ tests in the same run (for example, a downstream service was down), those failures are marked as an environment incident and excluded from flakiness scoring. Healthy tests don't get blamed for a bad night.

• Failures are clustered by root cause, not exact text. Messages are normalized (numbers, IDs, UUIDs, and quoted values stripped), so Timeout 15000ms and Timeout 30000ms on the same locator group together.

• Regressions are pinned to a commit. Related breakages that started in the same build are grouped as one incident with the first bad build, its commit, and the last good build.

• Flaky tests get a suggested fix based on the failure pattern: timeouts, detached/stale elements, or intermittent downstream dependencies.

• Performance drift detection compares early-window vs recent-window runtimes and flags tests that are degrading.

Works with your existing reports

Point it at any folder of JUnit-format XML. That covers most of the ecosystem:

• Java: Maven Surefire / Failsafe, TestNG, Gradle

• JavaScript/TypeScript: Playwright, Cypress, Jest (jest-junit), Vitest

• Python: pytest (--junitxml)

• CI: GitLab CI test report artifacts, Jenkins, GitHub Actions, Bamboo

Build metadata (build number, commit, branch, environment) is read from <properties> when present. Reports are re-read on every call, so new nightly artifacts show up without restarting the server.

Quick start

git clone https://github.com/Misfit1982/qa-metrics-mcp.git
cd qa-metrics-mcp
npm install          # also compiles TypeScript to dist/
npm run smoke        # end-to-end test over real MCP stdio

The repo ships with 14 nightly runs of synthetic data for a fictional e-commerce app, with known problems planted in it, so every tool has something to find out of the box.

Connect to Claude Code

claude mcp add qa-metrics -- node /absolute/path/to/qa-metrics-mcp/dist/server.js

Connect to Claude Desktop

Add to claude\_desktop\_config.json:

{
  "mcpServers": {
    "qa-metrics": {
      "command": "node",
      "args": \["/absolute/path/to/qa-metrics-mcp/dist/server.js"],
      "env": { "QA\_REPORTS\_DIR": "/absolute/path/to/your/junit-reports" }
    }
  }
}

Connect to Cursor

Add the same block to .cursor/mcp.json in your project (or the global Cursor MCP settings).

Configuration

Example session

**You:** Triage last night's run.

**Claude:** *(calls compare\_runs, detect\_regressions, find\_flaky\_tests, get\_failure\_clusters)*

**Summary:** Pass rate is 90% (27/30). Two real regressions, both pricing-related, started in build 1210. Three flaky tests are adding noise.

**Regressions:** Introduced in build **1210** (commit 9f3c2ab). Last good: 1209.

• ui.cart::apply\_coupon: cart total 89.99 vs expected 80.99 after SAVE10. *Draft bug: "Coupon discount not applied to cart total after pricing change in 9f3c2ab"*

• ui.checkout::tax\_calculation: order total off by about $9.72. Likely the same pricing change.

**Quarantine candidates:** order\_webhook (intermittent 504s from downstream), autocomplete (detached element), shipping\_options (locator timeout).

**For DevOps:** Build 1207 had an inventory-svc outage (ECONNREFUSED on 6 tests). Not a product defect.

See the full generated report in docs/sample-report.md.

Architecture

flowchart LR
  CI\[CI pipeline<br/>GitLab / Jenkins / Actions] -->|JUnit XML artifacts| DIR\[(Reports folder)]
  DIR --> P\[parser.ts<br/>JUnit → TestRun model]
  P --> M\[metrics.ts<br/>flakes · regressions · clusters · trends]
  M --> S\[server.ts<br/>11 MCP tools + triage prompt]
  M --> D\[dashboard.ts<br/>HTML export]
  S <-->|stdio / MCP| C\[Claude · Cursor · any MCP client]

src/
  parser.ts      JUnit XML → normalized TestRun / TestCase model
  metrics.ts     Pure analysis functions (no I/O, easy to unit-test)
  dashboard.ts   Self-contained HTML dashboard renderer
  server.ts      MCP server: tool definitions, input validation (zod), error handling
scripts/
  generate-sample-data.ts   Deterministic synthetic data with planted issues
  smoke-test.ts             End-to-end test via the official MCP client

Design choices:

• Read-only by default. Every analysis tool is annotated readOnlyHint. Only export\_dashboard writes a file, and only inside the output folder.

• Errors are tool errors, not crashes. A bad run ID returns a readable message the assistant can recover from.

• Pure metrics layer. All analysis is side-effect-free, so it can be reused in a CLI, a CI step, or a different server.

Testing

npm run smoke launches the compiled server exactly the way Claude Desktop or Cursor does (stdio, official MCP SDK client) and asserts that every planted issue is found:

  ✓ exposes 11 tools
  ✓ find\_flaky\_tests finds the 3 planted flakes and ignores the outage
  ✓ detect\_regressions pins both pricing breaks to build 1210 / commit 9f3c2ab
  ✓ get\_failure\_clusters flags the build 1207 outage as environmental
  ✓ get\_slowest\_tests flags guest\_checkout degradation
  ...
All 14 checks passed.

Roadmap

• Pull reports directly from the GitLab / GitHub Actions APIs

• Allure and TestNG native (testng-results.xml) formats

• Push the health report to Slack or Teams on a schedule

• Companion servers: bug-report drafting (failure → Jira ticket) and RCA assistant (failure cluster + commits → 5-Whys write-up)

About

Built by Jared Hood, QA leader with 20 years in SaaS, fintech, and media QA (Selenium, Playwright, Tosca, RestAssured, GitLab CI, AWS). I build AI-assisted QA tooling: MCP servers, test automation frameworks, and reporting that turns test results into decisions.

All data in this repo is synthetic and describes a fictional application.