testing-toolkit-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| TOOLKIT_HOME | No | Where downloaded jars and cached docs live. | ~/.testing-toolkit |
| TOOLKIT_JAVA | No | Java executable; Java 17 or newer is required. | java |
| TOOLKIT_REPO | No | Repository the tools run against when a tool call does not name one. | current directory |
| TOOLKIT_TQL_JAR | No | Path to a local tql-cli jar. | download |
| TOOLKIT_TQL_DOCS | No | Path to a local checkout's docs/rules directory. | download from GitHub |
| TOOLKIT_FLAKE_JAR | No | Path to a local flake-cli jar instead of the GitHub Release asset. | download |
| TOOLKIT_HTTP_HOST | No | Interface --http binds; can also be passed as --host. | 127.0.0.1 |
| TOOLKIT_HTTP_PORT | No | Port --http binds; can also be passed as --port. | 3000 |
| TOOLKIT_FLAKE_DOCS | No | Path to a local checkout's docs directory instead of GitHub. | download from GitHub |
| TOOLKIT_HTTP_TOKEN | No | Bearer token required by --http; can also be passed as --token. | none |
| TOOLKIT_TIMEOUT_MS | No | How long one tool invocation may run before it is killed. | 120000 |
| TOOLKIT_LLMEVAL_JAR | No | Path to a local llm-eval-harness CLI jar. | download |
| TOOLKIT_LLMEVAL_DOCS | No | Path to a local checkout's docs directory. | download from GitHub |
| TOOLKIT_TOOLS_VERSION | No | Release to download jars and docs from, when none is set locally. | latest |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": true
} |
| prompts | {
"listChanged": true
} |
| resources | {
"listChanged": true
} |
| completions | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| flake.rankA | Runs flake-detector's scorer over the ingested run history (.flake/history.db in the repository) and returns the most flaky tests first, each with a 0-1 score, run count, the signals behind the score and a plain-language explanation. Use it before retrying a failing test or deciding whether a red build is a regression. |
| flake.explainA | Returns the score of one test with every component that contributed to it (flip rate, rerun recovery, failure-message entropy, confidence interval, correlations with runner and hour) and the explanation flake-detector gives. Test ids look like com.acme.CheckoutTest#appliesCoupon. |
| flake.quarantine.listA | Reads the quarantine ledger (.flake/quarantine.yaml) and returns every entry with its reason, owner, expiry and whether it has expired. Expired entries fail the build until they are renewed or removed. |
| flake.gateA | Runs flake-detector's gate over a directory of JUnit XML reports (for example target/surefire-reports). Failures of quarantined or known-flaky tests do not block; any other failure does. Returns the verdict, each failure with the decision taken, and the reasoning. |
| tql.lintA | Runs test-quality-linter over test sources and returns findings with rule id, severity, location, message and fix hint. Catches tautological or constant assertions, tests with no assertion, assertions on stubs, hard-coded sleeps, swallowed exceptions, duplicate bodies and more. Paths are relative to the repository. |
| tql.rulesA | Returns the catalogue of rules the linter applies: id, name, default severity and a one-line description. Use tql.explain or the toolkit://tql/rules/{id} resource for the full documentation of one rule. |
| tql.explainA | Returns the documentation of one rule: what it catches, a bad example, a fixed example and its options. Rule ids look like TQL003. |
| llmeval.runA | Runs llm-eval-harness over a golden dataset with one prompt version and returns every case with its checks, score, tokens and latency, plus the pass rate. Mode replay (default) serves recorded responses and needs no API key; record calls the model and saves recordings. The result file path is returned for llmeval.drift. |
| llmeval.driftA | Compares two llm-eval-harness result files and returns the cases whose score dropped by more than the threshold or that flipped from pass to fail. Use it after llmeval.run to see whether a prompt change or model upgrade regressed anything. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
| review-test-quality | Lint the test sources with tql.lint, then summarise the findings by rule with concrete fixes, citing the rule documentation. |
| triage-red-build | Run flake.gate over the JUnit reports, explain each failure using the flakiness history, and say what is a regression and what is a known flake. |
| check-prompt-change | Run the golden set against the new prompt version with llmeval.run, compare the result with the baseline using llmeval.drift, and report regressed cases. |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
| flake-scoring | How the flakiness score is computed: flip rate, rerun recovery, failure-message entropy, Wilson interval and correlations, with a worked example. |
| README | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL001 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL002 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL003 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL004 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL005 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL006 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL007 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL008 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL009 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL010 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL011 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
| TQL012 | What each rule catches, a bad example, a fixed example and its options. README is the index. |
TDQS
Scored across 9 tools
Each tool is namespaced by subdomain (tql, flake, llmeval) and has a distinct responsibility: linting, rule docs, flake scoring/explanation/gating, and evaluation/drift comparison. There is no real overlap; even flake.rank and flake.gate are clearly separated as historical scoring vs build-gate decision.
Names follow a consistent lowercase dotted-prefix convention where the first segment identifies the subdomain (tql, flake, llmeval). Minor inconsistency comes from mixing verb segments (lint, run, explain) with noun segments (rules, gate) and one three-part name (flake.quarantine.list), but this remains predictable.
Nine tools is a well-scoped size for a testing toolkit covering three distinct areas: static test analysis, flaky-test detection/gating, and LLM evaluation. Each tool addresses a separate workflow step without bloat.
Core workflows are covered: listing/explaining rules and linting, scoring/explaining flaky tests and applying a gate, plus running evaluations and checking drift. The main gap is that quarantine entries can be listed but not added, renewed, or removed, so managing the quarantine lifecycle requires external edits.