playwright-e2e-mcp

Diagnose flaky test

diagnose-flaky

DestructiveIdempotent

Repeats failing Playwright tests with retries disabled to diagnose flakiness and return per-test evidence-based verdicts: consistently failing, flaky, or not reproducing.

Instructions

Repeat failing tests (Playwright or pytest-playwright) with retries disabled and return a per-test, evidence-based verdict: CONSISTENTLY FAILING (real bug), FLAKY (pass/fail counts, error-signature variance, duration spread) or NOT REPRODUCING. Runs all repetitions in one process by default. isolate: true also re-runs the failures with one worker to tell parallelism bugs (shared state) from flakiness inside the test; order: true (Playwright 1.64+) runs the files in random orders to catch tests that depend on what ran before them. Defaults to the tests that failed in the most recent run.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "diagnose-flaky"
  }
}

Input Schema

Name	Required	Description
`runs`	No	How many times to run the tests (default 3)
`order`	No	Also run the target files in `runs` random orders (Playwright 1.64+ --shuffle, one worker) to find tests that depend on what ran before them; returns the seed that reproduces it and the likely culprit test
`config`	No	playwright.config path or 1-based index when the project has several configs
`headed`	No	Run with a visible browser window
`browser`	No	Playwright project name to run
`isolate`	No	Also tell parallelism bugs from real flakiness: repeat in parallel (fully parallel, `workers` default 4), then repeat the failing tests with one worker
`workers`	No
`testFiles`	No	Tests to diagnose ("file:line" supported). Defaults to the tests that failed in the most recent run.
`timeoutMs`	No	Hard wall-clock limit per run (default 120000); one --repeat-each process gets this × runs
`projectRoot`	No	Project directory; defaults to the server working directory
`separateProcesses`	No	Start a fresh Playwright process per run instead of one --repeat-each process (slower; use when the flake depends on global setup or a cold start)

Output Schema

Name	Required	Description
`ok`	Yes	false when the call failed (see error) or the tests it ran failed
`error`	No
`tests`	No
`verdict`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed2 schema fields changedv0.13.0Nov 2, 2026
- removedInput schema / $schema
```
Removed value: -"https://json-schema.org/draft/2020-12/schema"
```
- removedInput schema / additionalProperties
```
Removed value: -{}
```

Changed2 schema fields changedv0.6.1Oct 27, 2026

addedInput schema / additionalProperties
```
Added value: +{}
```

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": false,
+  "properties": {
+    "error": {
+      "additionalProperties": false,
+      "properties": {
+        "hint": {
+          "type": "string"
+        },
+        "kind": {
+          "type": "string"
+        },
+        "message": {
+          "type": "string"
+        }
+      },
+      "required": [
+        "kind",
+        "message"
+      ],
+      "type": "object"
+    },
+    "ok": {
+      "description": "false when the call failed (see error) or the tests it ran failed",
+      "type": "boolean"
+    },
+    "tests": {
+      "items": {
+        "additionalProperties": false,
+        "properties": {
+          "failed": {
+            "maximum": 9007199254740991,
+            "minimum": -9007199254740991,
+            "type": "integer"
+          },
+          "file": {
+            "type": "string"
+          },
+          "line": {
+            "maximum": 9007199254740991,
+            "minimum": -9007199254740991,
+            "type": "integer"
+          },
+          "passed": {
+            "maximum": 9007199254740991,
+            "minimum": -9007199254740991,
+            "type": "integer"
+          },
+          "project": {
+            "type": "string"
+          },
+          "title": {
+            "type": "string"
+          },
+          "verdict": {
+            "enum": [
+              "NOT REPRODUCING",
+              "FLAKY",
+              "CONSISTENTLY FAILING",
+              "NO TESTS RAN"
+            ],
+            "type": "string"
+          }
+        },
+        "required": [
+          "file",
+          "title"
+        ],
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "verdict": {
+      "enum": [
+        "NOT REPRODUCING",
+        "FLAKY",
+        "CONSISTENTLY FAILING",
+        "NO TESTS RAN"
+      ],
+      "type": "string"
+    }
+  },
+  "required": [
+    "ok"
+  ],
+  "type": "object"
+}

Changed1 schema field changedv0.4.2Oct 23, 2026

addedInput schema / properties / order

Added value: +{
+  "description": "Also run the target files in `runs` random orders (Playwright 1.64+ --shuffle, one worker) to find tests that depend on what ran before them; returns the seed that reproduces it and the likely culprit test",
+  "type": "boolean"
+}

First observedv0.3.2Oct 20, 2026

TDQS

A4.1/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already flag destructive/openWorld, and the description adds substantive mechanics beyond them: one process by default, an `--repeat-each` process receiving `timeoutMs × runs`, an explicit slower cold-start path, and Playwright 1.64+ gating on `order`. It does not spell out what test side effects (databases, fixtures, cost) make this destructive, so a 4 rather than a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The verdict vocabulary and the default target are front-loaded, and each sentence carries information. It is still one long run-on paragraph with parenthetical stacking, and the `isolate`/`order` explanations largely duplicate the schema text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need no explanation, and an 11-parameter tool with zero required params is adequately covered by the described defaults. The only shortfall is that the destructive/potentially long-running nature of writing to a test project is not made explicit.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema is already at 91% description coverage, so most parameters explain themselves. The description adds the 'why' behind `isolate` and `order` (parallelism bugs vs. order dependence) but repeats the schema's own wording, and says nothing about `workers`, `browser`, `config`, or `projectRoot` beyond what the schema states.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and object ('repeat failing tests ... with retries disabled') and the exact output ('per-test, evidence-based verdict: CONSISTENTLY FAILING / FLAKY / NOT REPRODUCING'). That is clearly distinct from siblings such as bisect-failure (narrowing a commit) or run-test (a single execution) without needing to open a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a real default for lazy invocation ('defaults to the tests that failed in the most recent run') and explains the conditions that select the optional modes (isolate to separate shared-state bugs, order for order-dependence). It never names a sibling tool as an alternative or says when NOT to use it, which keeps it just below the top score.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.