Skip to main content
Glama

Diagnose flaky test

diagnose-flaky
DestructiveIdempotent

Repeat failing Playwright tests with retries disabled to return per-test verdicts—consistently failing, flaky, or not reproducing—and isolate parallelism bugs from real flakiness.

Instructions

Repeat failing tests with retries disabled and return a per-test, evidence-based verdict: CONSISTENTLY FAILING (real bug), FLAKY (pass/fail counts, error-signature variance, duration spread) or NOT REPRODUCING. Runs all repetitions in one process by default. isolate: true also re-runs the failures with one worker to tell parallelism bugs (shared state) from flakiness inside the test. Defaults to the tests that failed in the most recent run.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
runsNoHow many times to run the tests (default 3)
configNoplaywright.config path or 1-based index when the project has several configs
headedNoRun with a visible browser window
browserNoPlaywright project name to run
isolateNoAlso tell parallelism bugs from real flakiness: repeat in parallel (fully parallel, `workers` default 4), then repeat the failing tests with one worker
workersNo
testFilesNoTests to diagnose ("file:line" supported). Defaults to the tests that failed in the most recent run.
timeoutMsNoHard wall-clock limit per run (default 120000); one --repeat-each process gets this × runs
projectRootNoProject directory; defaults to the server working directory
separateProcessesNoStart a fresh Playwright process per run instead of one --repeat-each process (slower; use when the flake depends on global setup or a cold start)

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.3.2

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With annotations already declaring destructiveHint/readOnlyHint/idempotentHint, the description adds real operational context: repetitions run in one process by default, isolate changes the execution model, and the output taxonomy (CONSISTENTLY FAILING / FLAKY / NOT REPRODUCING) is disclosed. It never explains why destructiveHint=true (e.g., that it re-executes tests or touches state), leaving a small gap, but nothing contradicts the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the action and verdict taxonomy, then defaults and the isolate behavior. Dense but every sentence carries information; only the parenthetical elaboration on FLAKY evidence is slightly repetitive.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 10-parameter tool with no output schema, the description compensates well: it enumerates the possible verdicts and their evidence, documents the default test selection, and describes the two execution modes. An agent has enough to invoke and interpret results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is already ~90%, so 3 is the baseline, but the description goes beyond it by explaining the semantics of `isolate` as a diagnostic dichotomy (parallelism bug vs. in-test flakiness) rather than just repeating the schema text. `workers` is the one parameter with no description in either place, a minor gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource (repeat failing tests, retries disabled) plus the concrete output it produces (per-test verdicts). It is clearly distinguishable from siblings like run-test, get-failure, and analyze-history because it is the only one that re-runs failures to classify flakiness.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear context: operates on the tests that failed in the most recent run by default, and explains when to reach for `isolate` (to separate parallelism/shared-state bugs from true flakiness). It does not explicitly say when NOT to use it versus run-test or get-failure, so it falls just short of 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.