# testing-toolkit-mcp

[![CI](https://github.com/byreshb/testing-toolkit-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/byreshb/testing-toolkit-mcp/actions/workflows/ci.yml)

Coding agents now write and modify tests, but they have no view of test health. They retry
flaky tests instead of recognising them, write assertions that prove nothing, and change prompts
without any regression signal. Every one of those mistakes is invisible to the agent because the
tools that would catch it live outside the conversation.

This is a [Model Context Protocol](https://modelcontextprotocol.io) server, written in TypeScript
for the Node.js runtime, that gives agents that view as tools they can call. It wraps three Java
testing tools so any MCP client (Claude Code, Claude Desktop, IDE agents) can ask instead of
guess:

- [flake-detector](https://github.com/byreshb/flake-detector): which tests are flaky, why, and
  what is quarantined.
- [test-quality-linter](https://github.com/byreshb/test-quality-linter): lint a test file or a
  whole test tree and get findings with fix hints.
- [llm-eval-harness](https://github.com/byreshb/llm-eval-harness): run a golden set against a
  prompt version and report scores and drift.

## Requirements

- Node.js 22 or newer
- Java 17 or newer on the `PATH` (the tools are Java command-line jars)

## Install

The package is not published to npm yet (planned, see [docs/releasing.md](docs/releasing.md)).
Until then, build it from a checkout:

```bash
git clone https://github.com/byreshb/testing-toolkit-mcp.git
cd testing-toolkit-mcp
npm ci
npm run build
```

`node dist/index.js` then starts the server on stdio; `--version` and `--help` are the only
options.

## Quick start

Register the server with your MCP client, pointing `TOOLKIT_REPO` at the project whose tests you
care about. With Claude Code:

```bash
claude mcp add testing-toolkit -e TOOLKIT_REPO=/path/to/project \
  -- node /path/to/testing-toolkit-mcp/dist/index.js
```

With Claude Desktop or any other client that takes a JSON block:

```json
{
  "mcpServers": {
    "testing-toolkit": {
      "command": "node",
      "args": ["/path/to/testing-toolkit-mcp/dist/index.js"],
      "env": { "TOOLKIT_REPO": "/path/to/project" }
    }
  }
}
```

Then ask the agent about test health, or use one of the prompts. In Claude Code:

```text
> /mcp__testing-toolkit__triage-red-build target/surefire-reports
```

runs `flake.gate` over the JUnit reports, explains every failure from the run history, and ends
with the list of real regressions and the list of flakes to quarantine. Asking "review the tests
I just wrote" makes it call `tql.lint`, which returns findings with fix hints, and read resources
such as `toolkit://tql/rules/TQL003` to cite the rule it is applying. Before shipping a prompt
edit, `/mcp__testing-toolkit__check-prompt-change` runs the golden set with `llmeval.run` and
compares it with the baseline through `llmeval.drift`. A full walkthrough and the configuration
for other clients are in [docs/clients.md](docs/clients.md).

## Tools and resources

| Name                           | Kind     | What it does                                                     |
| ------------------------------ | -------- | ---------------------------------------------------------------- |
| `flake.rank`                   | tool     | Most flaky tests first, with scores and explanations             |
| `flake.explain`                | tool     | Why one test has the score it has                                |
| `flake.quarantine.list`        | tool     | Quarantined tests, expiry and owner                              |
| `flake.gate`                   | tool     | Should these failed tests block the build? Verdict and reasoning |
| `tql.lint`                     | tool     | Lint test sources; findings with rule, severity, location, fix   |
| `tql.rules`                    | tool     | Catalogue of linter rules                                        |
| `tql.explain`                  | tool     | Full documentation of one rule                                   |
| `llmeval.run`                  | tool     | Run a golden set against a prompt version; per-case scores       |
| `llmeval.drift`                | tool     | Cases that regressed between two runs                            |
| `toolkit://flake/scoring`      | resource | The scoring formulas, for citing                                 |
| `toolkit://tql/rules/{ruleId}` | resource | The rule's documentation page, for citing                        |
| `review-test-quality`          | prompt   | Lint, then summarise by rule with fixes                          |
| `triage-red-build`             | prompt   | Gate, then explain every failure                                 |
| `check-prompt-change`          | prompt   | Run the golden set, then drift against the baseline              |

Inputs, outputs and examples for every tool are in [docs/tools.md](docs/tools.md).

## Configuration

Everything is read from environment variables, so it works the same in an MCP client's config
block, a shell and a container.

| Variable                | Default              | Meaning                                                              |
| ----------------------- | -------------------- | -------------------------------------------------------------------- |
| `TOOLKIT_REPO`          | current directory    | Repository the tools run against when a tool call does not name one. |
| `TOOLKIT_HOME`          | `~/.testing-toolkit` | Where downloaded jars and cached docs live.                          |
| `TOOLKIT_JAVA`          | `java`               | Java executable; Java 17 or newer is required.                       |
| `TOOLKIT_FLAKE_JAR`     | download             | Path to a local `flake-cli` jar instead of the GitHub Release asset. |
| `TOOLKIT_TQL_JAR`       | download             | Path to a local `tql-cli` jar.                                       |
| `TOOLKIT_LLMEVAL_JAR`   | download             | Path to a local `llm-eval-harness` CLI jar.                          |
| `TOOLKIT_FLAKE_DOCS`    | download from GitHub | Path to a local checkout's `docs` directory instead of GitHub.       |
| `TOOLKIT_TQL_DOCS`      | download from GitHub | Path to a local checkout's `docs/rules` directory.                   |
| `TOOLKIT_LLMEVAL_DOCS`  | download from GitHub | Path to a local checkout's `docs` directory.                         |
| `TOOLKIT_TOOLS_VERSION` | `latest`             | Release to download jars and docs from, when none is set locally.    |
| `TOOLKIT_TIMEOUT_MS`    | `120000`             | How long one tool invocation may run before it is killed.            |
| `TOOLKIT_HTTP_TOKEN`    | none                 | Bearer token required by `--http`; can also be passed as `--token`.  |
| `TOOLKIT_HTTP_HOST`     | `127.0.0.1`          | Interface `--http` binds; can also be passed as `--host`.            |
| `TOOLKIT_HTTP_PORT`     | `3000`               | Port `--http` binds; can also be passed as `--port`.                 |

On first use of each tool family the server looks for a jar in `TOOLKIT_HOME/jars/`, and when
there is none it downloads the CLI asset of the matching GitHub Release. Delete that directory
to pick up a newer release, or pin one with `TOOLKIT_TOOLS_VERSION`. If Java is missing or older
than 17 every tool call fails with a message that says how to fix it.

The server speaks stdio by default; `--http` serves Streamable HTTP instead, for sharing one
running server between clients or for a container. See
[docs/clients.md](docs/clients.md#remote-clients-streamable-http) for the HTTP transport and
[docs/clients.md](docs/clients.md#docker) for running it in Docker.

## Building and testing

```bash
npm ci              # install the pinned dependencies
npm run check       # tsc --noEmit, eslint and prettier --check
npm test            # vitest with coverage (fails under 85% lines)
npm run build       # compile to dist/
npm run format      # apply prettier
```

### Continuous integration

Every push and pull request runs the GitHub Actions workflow in `.github/workflows/ci.yml`:
`npm ci`, `npm run check` and `npm test`, with the coverage report uploaded as an artifact.

### Releasing

1. Move the `Unreleased` notes in `CHANGELOG.md` under a new version heading and set that
   version in `package.json`.
2. Commit, then tag and push: `git tag -a v1.2.3 -m "Release 1.2.3" && git push origin v1.2.3`.
3. The release workflow in `.github/workflows/release.yml` checks the tag matches
   `package.json`, builds and packs the package, and publishes a GitHub Release with the changelog
   section as its notes and the tarball attached.

Full steps, including the planned but not yet configured npm publishing, are in
[docs/releasing.md](docs/releasing.md).

## License

Apache License 2.0, see [LICENSE](LICENSE).