Skip to main content
Glama
byreshb

testing-toolkit-mcp

by byreshb

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault
TOOLKIT_HOMENoWhere downloaded jars and cached docs live.~/.testing-toolkit
TOOLKIT_JAVANoJava executable; Java 17 or newer is required.java
TOOLKIT_REPONoRepository the tools run against when a tool call does not name one.current directory
TOOLKIT_TQL_JARNoPath to a local tql-cli jar.download
TOOLKIT_TQL_DOCSNoPath to a local checkout's docs/rules directory.download from GitHub
TOOLKIT_FLAKE_JARNoPath to a local flake-cli jar instead of the GitHub Release asset.download
TOOLKIT_HTTP_HOSTNoInterface --http binds; can also be passed as --host.127.0.0.1
TOOLKIT_HTTP_PORTNoPort --http binds; can also be passed as --port.3000
TOOLKIT_FLAKE_DOCSNoPath to a local checkout's docs directory instead of GitHub.download from GitHub
TOOLKIT_HTTP_TOKENNoBearer token required by --http; can also be passed as --token.none
TOOLKIT_TIMEOUT_MSNoHow long one tool invocation may run before it is killed.120000
TOOLKIT_LLMEVAL_JARNoPath to a local llm-eval-harness CLI jar.download
TOOLKIT_LLMEVAL_DOCSNoPath to a local checkout's docs directory.download from GitHub
TOOLKIT_TOOLS_VERSIONNoRelease to download jars and docs from, when none is set locally.latest

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

This server publishes no instructions, or was last inspected before Glama recorded them.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": true
}
prompts
{
  "listChanged": true
}
resources
{
  "listChanged": true
}
completions
{}

Tools

Functions exposed to the LLM to take actions

NameDescription
flake.rankA

Runs flake-detector's scorer over the ingested run history (.flake/history.db in the repository) and returns the most flaky tests first, each with a 0-1 score, run count, the signals behind the score and a plain-language explanation. Use it before retrying a failing test or deciding whether a red build is a regression.

flake.explainA

Returns the score of one test with every component that contributed to it (flip rate, rerun recovery, failure-message entropy, confidence interval, correlations with runner and hour) and the explanation flake-detector gives. Test ids look like com.acme.CheckoutTest#appliesCoupon.

flake.quarantine.listA

Reads the quarantine ledger (.flake/quarantine.yaml) and returns every entry with its reason, owner, expiry and whether it has expired. Expired entries fail the build until they are renewed or removed.

flake.gateA

Runs flake-detector's gate over a directory of JUnit XML reports (for example target/surefire-reports). Failures of quarantined or known-flaky tests do not block; any other failure does. Returns the verdict, each failure with the decision taken, and the reasoning.

tql.lintA

Runs test-quality-linter over test sources and returns findings with rule id, severity, location, message and fix hint. Catches tautological or constant assertions, tests with no assertion, assertions on stubs, hard-coded sleeps, swallowed exceptions, duplicate bodies and more. Paths are relative to the repository.

tql.rulesA

Returns the catalogue of rules the linter applies: id, name, default severity and a one-line description. Use tql.explain or the toolkit://tql/rules/{id} resource for the full documentation of one rule.

tql.explainA

Returns the documentation of one rule: what it catches, a bad example, a fixed example and its options. Rule ids look like TQL003.

llmeval.runA

Runs llm-eval-harness over a golden dataset with one prompt version and returns every case with its checks, score, tokens and latency, plus the pass rate. Mode replay (default) serves recorded responses and needs no API key; record calls the model and saves recordings. The result file path is returned for llmeval.drift.

llmeval.driftA

Compares two llm-eval-harness result files and returns the cases whose score dropped by more than the threshold or that flipped from pass to fail. Use it after llmeval.run to see whether a prompt change or model upgrade regressed anything.

Prompts

Interactive templates invoked by user choice

NameDescription
review-test-qualityLint the test sources with tql.lint, then summarise the findings by rule with concrete fixes, citing the rule documentation.
triage-red-buildRun flake.gate over the JUnit reports, explain each failure using the flakiness history, and say what is a regression and what is a known flake.
check-prompt-changeRun the golden set against the new prompt version with llmeval.run, compare the result with the baseline using llmeval.drift, and report regressed cases.

Resources

Contextual data attached and managed by the client

NameDescription
flake-scoringHow the flakiness score is computed: flip rate, rerun recovery, failure-message entropy, Wilson interval and correlations, with a worked example.
READMEWhat each rule catches, a bad example, a fixed example and its options. README is the index.
TQL001What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL002What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL003What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL004What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL005What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL006What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL007What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL008What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL009What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL010What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL011What each rule catches, a bad example, a fixed example and its options. README is the index.
TQL012What each rule catches, a bad example, a fixed example and its options. README is the index.

TDQS

A4.2/5.0

Scored across 9 tools

Disambiguation5/5

Each tool is namespaced by subdomain (tql, flake, llmeval) and has a distinct responsibility: linting, rule docs, flake scoring/explanation/gating, and evaluation/drift comparison. There is no real overlap; even flake.rank and flake.gate are clearly separated as historical scoring vs build-gate decision.

Naming Consistency4/5

Names follow a consistent lowercase dotted-prefix convention where the first segment identifies the subdomain (tql, flake, llmeval). Minor inconsistency comes from mixing verb segments (lint, run, explain) with noun segments (rules, gate) and one three-part name (flake.quarantine.list), but this remains predictable.

Tool Count5/5

Nine tools is a well-scoped size for a testing toolkit covering three distinct areas: static test analysis, flaky-test detection/gating, and LLM evaluation. Each tool addresses a separate workflow step without bloat.

Completeness4/5

Core workflows are covered: listing/explaining rules and linting, scoring/explaining flaky tests and applying a gate, plus running evaluations and checking drift. The main gap is that quarantine entries can be listed but not added, renewed, or removed, so managing the quarantine lifecycle requires external edits.

Maintenance

ActivityMaintained
ResponsivenessNo issues