It is an MCP server that lets an LLM agent analyze Playwright test reports, focusing on failures, reliability, and history.
Get a summary of the latest test run (counts, duration, start time).
Get failed tests with details like test ID, file, project, status, and error message.
Group failures by error patterns (e.g., locator-not-found, assertion-failure).
Identify the most unstable tests (failing or flaking most often).
View the run-by-run history of a specific test, including failure modes and current status.
Point to your own Playwright JSON report and history directory via environment variables.
The server handles both latest-run tools and history tools using the same parser.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Playwright Report MCP What failed in the latest test run?"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
An MCP server that lets an LLM agent read Playwright test reports. A Playwright JSON report is tens of thousands of lines and mostly noise; these tools answer the questions you would otherwise scroll for — what failed, which failures share a cause, and whether a test has been failing for weeks or only started today.
> which tests are the least reliable?
rate test last pattern
1.00 checkout.spec.ts > TC-C02: order summary failed locator-ambiguous
1.00 dashboard.spec.ts > TC-D01: loads widgets failed timing-or-waiting
0.80 cart.spec.ts > TC-R01: persists the cart flaky assertion-failureTool | Answers |
| How the latest run went: counts, duration, plus |
| The failed tests, with |
| Those failures grouped by what their errors look like — |
| The tests that fail or flake most often across past runs, worst first. Where an investigation starts, since every other history question needs a |
| One test, run by run: how often it failed, how it failed each time, and whether it is still failing. |
Two details worth knowing: a test running on two browsers produces two results
per run, so total_runs and total_results are counted separately and rates
are out of the latter. And instability_rate counts flakes alongside failures,
because a test that only ever passes on a retry is not a healthy test.
Related MCP server: deploytest MCP server
Python ≥ 3.10
uv — brew install uv, or curl -LsSf https://astral.sh/uv/install.sh | sh
Node.js, only for mcp dev (the MCP Inspector)
uv syncThat is the whole setup. A sample report and five runs of history are bundled in
data/, so every tool answers immediately — no configuration, and no Playwright
project needed to try it.
The repository ships a project-scoped .mcp.json, so there is nothing to write:
claude # from the repository root, then approve the server when promptedCheck it with /mcp in the session, or claude mcp list in a terminal, then
ask: "which tests are the least reliable?"
Servers load at startup, so restart the session if it was already running. To
register it globally instead, pass --directory so uv finds this project's
venv from anywhere:
claude mcp add playwright-report -s user -- \
uv --directory /absolute/path/to/playwright-test-analysis-mcp run playwright-report-mcpAdd to claude_desktop_config.json (~/Library/Application Support/Claude/ on
macOS), then quit with Cmd-Q and reopen:
{
"mcpServers": {
"playwright-report": {
"command": "uv",
"args": ["--directory", "/absolute/path/to/playwright-test-analysis-mcp",
"run", "playwright-report-mcp"]
}
}
}Logs land in ~/Library/Logs/Claude/mcp-server-playwright-report.log.
uv run mcp dev src/playwright_report_mcp/server.py # Inspector, in a browser
uv run playwright-report-mcp # plain stdio serverGenerate a report with the json reporter:
npx playwright test --reporter=json > results.jsonThen set two variables — REPORT_PATH for the latest run, HISTORY_DIR for a
directory of past ones named run-001.json, run-002.json, … Both default into
data/, both are re-read on every call, and relative paths resolve from the
repository root rather than the working directory.
Build history by copying each run in as the next number; anything matching
run-*.json is read. Three runs is where a trend starts to mean anything.
cp results.json data/history/run-006.jsonWhere you set the two variables depends on how the server was started:
Launched by | Configuration comes from |
Claude Code / Desktop | the |
Your shell ( | exported variables |
MCP Inspector ( | the Inspector's own Environment Variables fields |
The Inspector is the odd one out: it spawns servers with a fixed set of
variables (HOME, LOGNAME, PATH, SHELL, TERM, USER) and drops
everything else, so exporting REPORT_PATH before mcp dev has no effect.
uv run pytest58 tests, no mocking — every one reads a real report through the real parser.
File | Covers | |
| 20 | Parsing one report: stats, failure fields, flaky tests, ANSI stripping, pattern grouping, and the messages given for a missing, truncated or non-Playwright file |
| 26 | Aggregating many reports: counts and rates, run-vs-result counting, the instability ranking and its tie-breaks, ordering, empty and unknown cases |
| 7 | The MCP contract — each tool listed, called over an in-memory client, returning the expected shape |
| 5 | Path resolution: relative vs absolute, defaults, explicit overrides |
Fixtures in tests/fixtures/ hold the reports they read, including a run with
flaky tests and a run that died in global setup. Tests needing exact rates build
their runs in a tmp_path instead.
src/playwright_report_mcp/
server.py the five MCP tools, and nothing else
config.py where REPORT_PATH and HISTORY_DIR resolve to
playwright_report.py the one place that knows Playwright's JSON shape
models/ the normalized shapes every analysis starts from
analysis/ failure classification, history, instability ranking
history/loader.py reads the stored runs, reusing playwright_report.py
data/
results.json sample report: the latest run
history/run-00N.json five past runs, for the history toolsA stored run and the latest run go through the same parser, so the history tools and the latest-run tools cannot disagree about what a test is.
Symptom | Cause |
Tools answer about the wrong tests |
|
| The path in the message is what |
| Not |
| A client ran |
Server missing from | Servers load at startup: restart the session, and approve the project-scoped server |
| Node.js is missing, so there is no |
Avoid uv run mcp install …: it registers the server with --with mcp[cli] in
an isolated environment and no --directory, so this package is never
importable.
Return failed tests of the latest Playwright test run.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It conveys read-only behavior via 'Return' and adds the useful 'latest run' scope, but it does not disclose behavior such as how empty results are handled or whether failures are ordered or grouped. These gaps are minor for a simple parameterless read and are partly covered by the output schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence that front-loads the action and object and contains no filler. Every word contributes to selection and invocation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the zero-parameter complexity and the presence of an output schema, the description is nearly complete for correct tool selection and invocation. The only material gap is the lack of any interaction with the sibling get_test_summary, but that is a usage-guidance concern rather than a completeness failure for this simple read.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there is no parameter semantics burden for the description to carry. The baseline of 4 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear action ('Return'), a precise resource ('failed tests'), and a scoping qualifier ('latest Playwright test run'). It effectively distinguishes itself from get_test_summary by focusing on failures rather than a summary.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no explicit guidance about when to use this tool versus get_test_summary. It implies only that results are scoped to the latest run, but does not state when to prefer failures over a summary or whether get_test_summary is the alternative for successful tests.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return summary of the latest Playwright test run.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| flaky | Yes | |
| failed | Yes | |
| passed | Yes | |
| skipped | Yes | |
| start_time | Yes | |
| duration_ms | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral disclosure burden. The word 'Return' suggests a read-only operation, but the description does not mention what happens when no test run exists, whether the summary is limited by filters, or any side effects. This is minimally adequate but lacks richer behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, compact sentence that front-loads the action ('Return') and the object ('summary') with no filler or unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter retrieval tool, the description is sufficiently complete: the resource is identified, the action is clear, and the presence of an output schema covers return-value details. The main gap is lack of usage guidance relative to the sibling, but that is a separate dimension.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the input schema is trivially complete. The description does not need to explain parameter semantics, and the baseline of 4 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Return') and the resource ('summary of the latest Playwright test run'). It distinguishes from the sibling 'get_failures' by focusing on the overall summary rather than failures, but it does not explicitly name or contrast the sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus 'get_failures' or any other alternative. The description implies it is for retrieving a summary, but it does not state exclusion criteria, prerequisites, or when another tool would be preferable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0get_failuresget_test_summaryScored across 2 tools
get_failures and get_test_summary have clearly distinct purposes: one retrieves only failed tests, the other an aggregate summary. There is no overlap or ambiguity between them.
Both tools follow a consistent get_[noun] pattern, making the naming predictable and easy to infer. The convention is uniform across the entire set.
With only two tools, the set is on the thin side and falls into the borderline range. The tools are focused and justified, but the count is minimal for a server that could reasonably include more report-oriented operations.
For a server scoped to the latest Playwright test run, get_failures and get_test_summary cover the primary reporting needs. Minor gaps exist—such as retrieving test details or run history—but they do not create dead ends for the core use case.
Direct access to Cypress tests results and accessibility reports in your AI workflow.
Discover Playwright workflows, start runs, and inspect results in Playrunner Cloud.
Ingest and search LogsLoom logs from coding agents.
Agentic testing: HyperExecute jobs, test failure triage, SmartUI visual diffs, a11y audits
