testing-toolkit-mcp

Coding agents now write and modify tests, but they have no view of test health. They retry flaky tests instead of recognising them, write assertions that prove nothing, and change prompts without any regression signal. Every one of those mistakes is invisible to the agent because the tools that would catch it live outside the conversation.

This is a Model Context Protocol server, written in TypeScript for the Node.js runtime, that gives agents that view as tools they can call. It wraps three Java testing tools so any MCP client (Claude Code, Claude Desktop, IDE agents) can ask instead of guess:

• flake-detector: which tests are flaky, why, and what is quarantined.

• test-quality-linter: lint a test file or a whole test tree and get findings with fix hints.

• llm-eval-harness: run a golden set against a prompt version and report scores and drift.

Requirements

• Node.js 22 or newer

• Java 17 or newer on the PATH (the tools are Java command-line jars)

Related MCP server: Autonomous QA Engineer MCP

Install

The package is not published to npm yet (planned, see docs/releasing.md). Until then, build it from a checkout:

git clone https://github.com/byreshb/testing-toolkit-mcp.git
cd testing-toolkit-mcp
npm ci
npm run build

node dist/index.js then starts the server on stdio; --version and --help are the only options.

Quick start

Register the server with your MCP client, pointing TOOLKIT_REPO at the project whose tests you care about. With Claude Code:

claude mcp add testing-toolkit -e TOOLKIT_REPO=/path/to/project \
  -- node /path/to/testing-toolkit-mcp/dist/index.js

With Claude Desktop or any other client that takes a JSON block:

{
  "mcpServers": {
    "testing-toolkit": {
      "command": "node",
      "args": ["/path/to/testing-toolkit-mcp/dist/index.js"],
      "env": { "TOOLKIT_REPO": "/path/to/project" }
    }
  }
}

Then ask the agent about test health, or use one of the prompts. In Claude Code:

> /mcp__testing-toolkit__triage-red-build target/surefire-reports

runs flake.gate over the JUnit reports, explains every failure from the run history, and ends with the list of real regressions and the list of flakes to quarantine. Asking "review the tests I just wrote" makes it call tql.lint, which returns findings with fix hints, and read resources such as toolkit://tql/rules/TQL003 to cite the rule it is applying. Before shipping a prompt edit, /mcp__testing-toolkit__check-prompt-change runs the golden set with llmeval.run and compares it with the baseline through llmeval.drift. A full walkthrough and the configuration for other clients are in docs/clients.md.

Tools and resources

Inputs, outputs and examples for every tool are in docs/tools.md.

Configuration

Everything is read from environment variables, so it works the same in an MCP client's config block, a shell and a container.

On first use of each tool family the server looks for a jar in TOOLKIT_HOME/jars/, and when there is none it downloads the CLI asset of the matching GitHub Release. Delete that directory to pick up a newer release, or pin one with TOOLKIT_TOOLS_VERSION. If Java is missing or older than 17 every tool call fails with a message that says how to fix it.

The server speaks stdio by default; --http serves Streamable HTTP instead, for sharing one running server between clients or for a container. See docs/clients.md for the HTTP transport and docs/clients.md for running it in Docker.

Building and testing

npm ci              # install the pinned dependencies
npm run check       # tsc --noEmit, eslint and prettier --check
npm test            # vitest with coverage (fails under 85% lines)
npm run build       # compile to dist/
npm run format      # apply prettier

Continuous integration

Every push and pull request runs the GitHub Actions workflow in .github/workflows/ci.yml: npm ci, npm run check and npm test, with the coverage report uploaded as an artifact.

Releasing

1. Move the Unreleased notes in CHANGELOG.md under a new version heading and set that version in package.json.

2. Commit, then tag and push: git tag -a v1.2.3 -m "Release 1.2.3" && git push origin v1.2.3.

3. The release workflow in .github/workflows/release.yml checks the tag matches package.json, builds and packs the package, and publishes a GitHub Release with the changelog section as its notes and the tarball attached.

Full steps, including the planned but not yet configured npm publishing, are in docs/releasing.md.

License

Apache License 2.0, see LICENSE.