Skip to main content
Glama

Diagnose flaky test

diagnose-flaky
DestructiveIdempotent

Rerun failing Playwright tests multiple times with retries disabled to return a verdict: consistently failing, flaky, or not reproducing.

Instructions

Run a failing test multiple times with retries disabled and return an evidence-based verdict: CONSISTENTLY FAILING (real bug), FLAKY (intermittent — includes pass/fail counts and error-signature variance) or NOT REPRODUCING (one-off). Defaults to the tests that failed in the most recent run.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
runsNoHow many times to run the tests (default 3)
configNoplaywright.config path or 1-based index when the project has several configs
headedNoRun with a visible browser window
browserNoPlaywright project name to run
workersNo
testFilesNoTests to diagnose ("file:line" supported). Defaults to the tests that failed in the most recent run.
timeoutMsNoHard wall-clock limit per run (default 120000)
projectRootNoProject directory; defaults to the server working directory

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.2.2

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds behavior the annotations don't convey: retries are disabled during diagnosis, tests are executed multiple times, and the result varies in the described way (pass/fail counts, error-signature variance). Annotations cover safety (destructive/idempotent/openWorld), so the description is free to focus on run semantics, which it does well.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One dense sentence, front-loaded with the action and the three possible verdicts. Every clause carries information; nothing is padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 8-parameter tool with no output schema, the description covers the output shape (verdict categories and what each contains) and the default targeting, which is the critical missing piece. Remaining parameters are handled by the schema, so the gap is minor.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 88% and already documents runs, testFiles, timeoutMs, and config. The description repeats the testFiles and runs defaults without adding format or interaction details, so it does not materially extend the schema. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (run failing test repeatedly), the key constraint (retries disabled), and the output (evidence-based verdict with three named outcomes). An agent can distinguish it from run-test purely from this description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clear context: use when a test is failing and you need to know whether it's a real bug, flaky, or a one-off, and it defaults to the most recent failure set. It doesn't explicitly name a sibling to use instead for plain execution, but the diagnostic framing makes the boundary with run-test obvious.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.