detect_flaky_tests
Identify flaky tests by analyzing historical pass/fail data. Input test history with names and statuses to detect inconsistent test results.
Instructions
Detect flaky tests from historical pass/fail data.
Each item should contain test_name and status.Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| test_history | Yes |
Implementation Reference
- testFailureAnalysis.py:69-97 (handler)The MCP tool handler that detects flaky tests from historical pass/fail data. It aggregates pass/fail counts per test, and flags tests with >=5 runs where both pass and fail counts are >0, returning their failure rates.
@mcp.tool() def detect_flaky_tests(test_history: list[dict]) -> dict: """ Detect flaky tests from historical pass/fail data. Each item should contain test_name and status. """ stats = defaultdict(lambda: {"pass": 0, "fail": 0}) for run in test_history: name = run["test_name"] status = run["status"].lower() if status in stats[name]: stats[name][status] += 1 flaky = [] for test_name, result in stats.items(): total = result["pass"] + result["fail"] if total >= 5 and result["pass"] > 0 and result["fail"] > 0: failure_rate = result["fail"] / total flaky.append({ "test_name": test_name, "pass_count": result["pass"], "fail_count": result["fail"], "failure_rate": round(failure_rate, 2) }) return {"flaky_tests": flaky} - testFailureAnalysis.py:69-70 (registration)The @mcp.tool() decorator registers detect_flaky_tests as an MCP tool on the FastMCP server instance.
@mcp.tool() def detect_flaky_tests(test_history: list[dict]) -> dict: