Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@qa-toolkit-mcp compare runs search-25 and search-26"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
An MCP server that reads test reports and turns them into a regression analysis a model can work with. It does not run your tests. It reads the reports your test runs already write, and answers questions like "what regressed between Monday and Friday, and which of these failures are just the same known issues?"
I built it so an agent can do the boring part of a weekly regression review for me: line up the runs, compare them, and tell me what actually broke instead of handing me a raw diff.
You point it at a folder of run reports (the JSON your CI, your cron job, or your manual runs already produce) and it gives a model three tools, one resource, and one prompt:
qa_list_runs - lists the runs in the folder. You can filter by suite or by date, and it pages.
qa_get_run - returns one run. By default it only lists the failures, to keep the model's context small. Pass include_passed when you want everything.
qa_compare_runs - takes two runs and sorts what changed between them. This is the main one.
run://{run_id}/summary.md (resource) - a Markdown summary of a single run the host can load.
weekly_regression_review (prompt) - walks the model through a week: list the runs, compare each consecutive pair, and write a short report.
qa_compare_runs does not hand you a raw diff. It sorts every test into the buckets a QA person actually acts on:
regression - passed in A, failed in B. The one you care about most.
fix - failed in A, passed in B.
persistent failure - failed in both. It also tells you whether it is the same error as before or a new one (see fingerprints below).
new test / removed test - showed up or disappeared between the two runs.
classification change - the QA label changed, even when the pass/fail did not. "Still failing here, but we changed our mind about why."
Here is what the Markdown output looks like:
# Compare `search-25` → `search-26`
Suite: search → search
Started: 2026-05-25T09:00:00 → 2026-05-26T09:00:00
**1 regression(s) · 1 fix(es) · 2 persistent · 0 new · 0 removed · 1 reclassified**
## Regressions (passed → failed)
- `SI-POS-008` — **AssertionError**: Expected success, got 'error' - timeout
## Fixes (failed → passed)
- `SI-POS-005`
## Persistent failures
- `SI-POS-006` (same error) — SQLGrammarException: unexpected token
- `SI-POS-007` (same error) — SQLGrammarException: unexpected token
## Classification changes (QA oracle changed its mind)
- `SI-POS-007`: unclassified → bug realEvery tool can return JSON instead of Markdown, for when the agent needs to read the numbers and not the prose.
Related MCP server: ReAI MCP Server
It does not run your tests. It only reads reports. Whatever writes the report (a pytest job, a pipeline, a person) stays separate from this server.
It does not compute the fingerprint that groups "the same error". The producer of the report does that (more on why below).
No HTTP yet. It speaks stdio only, which is what local MCP clients use.
No dashboard and no UI. That is a separate thing that will read the same JSON one day.
It is not on PyPI yet. You install it from the repo.
Two report formats, detected per file. Drop either kind into the runs folder and the server works out which one it is:
Native reports have a schema_version field and are checked against schemas/run-report.v1.json.
Classification reports are what a QA pipeline tends to write: a list of the failures, each with a human-assigned label. If there is a JUnit XML file sitting next to it with the same name, the server reads that too, so now it also knows about the tests that passed.
is_exhaustive. A native report, or a classification report with its XML sibling, knows every test that ran. A bare classification report only knows the failures. The compare keeps track of this: when one side only lists failures, a test that is missing is treated as passed. So a test that failed in A and is gone in B counts as a fix, not as a removed test.
Fingerprints. Each failure in a report carries a fingerprint: a hash of the test id, the error type, and the normalized message. Two failures with the same fingerprint are treated as the same root cause. That is how qa_compare_runs tells "still the same bug" apart from "now it fails for a different reason".
Flat parameters. The tools take plain top-level arguments (run_a, run_b, and so on), not one nested params object. There is a story behind that, in the next section.
Configuration. Copy .env.example to .env and set QA_TOOLKIT_RUNS_DIR to your reports folder. A real env var wins over .env, which wins over the default of ./runs/. .env is gitignored, so it stays one per machine.
It reads reports instead of running tests because the thing that runs the tests and the thing that reads them should not be glued together. CI runs the tests on its own schedule and writes a report. This server reads that report whenever an agent asks. Anything that writes the schema can be read, and a dashboard could read the same files later without sharing a line of code with this server.
It categorizes instead of diffing because a raw diff makes you read everything again. A QA engineer does not treat all changes the same: a regression is urgent, a known persistent failure is something you already triaged. So the tool does that sorting up front.
The fingerprint lives in the producer, not here. Error messages change every run (timestamps, ids, line numbers), so comparing raw messages makes the same bug look new every time. A fingerprint stays stable. I put it in the producer because only the producer knows which parts of its own messages are the volatile bits, and that keeps this server framework-agnostic.
The parameters are flat because models are bad at the nested version. I found this with my own test harness: a local model could only call qa_compare_runs about one time in ten, because FastMCP was wrapping every argument under a required params object and the model kept sending the arguments flat. So I flattened the schema to match what models actually send. The whole investigation is written up in ADR 0001.
Install it:
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -e ".[dev]"The server speaks stdio, so you do not start it by hand. Your MCP client launches it as a subprocess. Register it and point it at your reports.
Claude Code:
claude mcp add qa-toolkit -s user -- `
"<repo>\.venv\Scripts\python.exe" -m qa_toolkit_mcp.serverClaude Desktop, in claude_desktop_config.json:
{
"mcpServers": {
"qa-toolkit": {
"command": "<repo>\\.venv\\Scripts\\python.exe",
"args": ["-m", "qa_toolkit_mcp.server"]
}
}
}Or poke at it by hand with the MCP Inspector:
npx @modelcontextprotocol/inspector .\.venv\Scripts\python.exe -m qa_toolkit_mcp.serverRun the tests with pytest. The suite is layered the way I test things: the pure functions (compare, storage, the adapter, the formatters) on their own, then the tools through their real entry points, and then a set of metamorphic checks on qa_compare_runs - properties that have to hold whatever the input. For example, comparing a run against itself reports nothing changed, and the regressions going from A to B are exactly the fixes going from B to A.
qa_toolkit_mcp/
server.py the MCP server: the tools, the resource, the prompt, the entry point
models.py Pydantic models for the report schema
storage.py reads files, keeps paths safe, detects the format
adapter_classification.py turns a classification report (+ JUnit XML) into the canonical model
compare.py the regression analysis, pure functions, no I/O
formatters.py models to Markdown or JSON
config.py .env and env-var handling
schemas/
run-report.v1.json the report contract, the source of truth
docs/adr/ decision records (0001 - why the tool parameters are flat)
evaluations/ eval questions for the server
tests/ the layered + metamorphic suiteMIT, see LICENSE.
Compare two test runs and categorize the differences.
`run_a` is treated as baseline (older), `run_b` as newer.
Categories returned:
regressions passed in A, failed/error in B (highest priority)
fixes failed/error in A, passed in B
persistent_failures failed in both
same_error fingerprints match → same root cause
different_error fingerprints differ → root cause changed
new_tests in B but not A
removed_tests in A but not B
other_changes transitions involving skipped (low priority)
Flakiness detection requires N>2 runs and is not in this tool. Use the
weekly_regression_review prompt to orchestrate multi-run analysis.
Returns:
Markdown summary or JSON of the full ComparisonResult model.
Error response: string starting with "Error: ...".
| Name | Required | Description | Default |
|---|---|---|---|
| run_a | Yes | Baseline run_id (treated as 'before'). | |
| run_b | Yes | Newer run_id (treated as 'after'). | |
| response_format | No | 'markdown' for human-readable, 'json' for programmatic. | markdown |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false. The description adds detailed behavioral context: categorization logic with priority ordering, that error responses start with 'Error:', and the return formats. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is well-structured with bullet points and sections, each sentence adds value. It is concise yet comprehensive, front-loading the purpose and then detailing categories.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with moderate complexity (multi-category comparison), the description covers the full logic, return formats (Markdown/JSON), error handling, and even mentions what is not covered (flakiness). No output schema provided but the description explains what is returned.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds context for run_a and run_b as baseline/newer, reinforcing schema statements, and explains how the parameters drive the categorization logic, which adds meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it compares two test runs and categorizes differences, listing all categories. It distinguishes itself from sibling tools (qa_get_run, qa_list_runs) by focusing on comparison rather than retrieval or listing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly explains that run_a is baseline (older) and run_b is newer, provides guidance on flakiness detection requiring N>2 runs and directs to an alternative (weekly_regression_review prompt). This covers when to use and when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return a single test run by id.
By default, only failed/error tests are listed in the body (to keep context
small). Set `include_passed=true` for the full inventory.
Returns:
Markdown or JSON depending on response_format. JSON returns the full
RunReport model conforming to schemas/run-report.v1.json.
Error response: string starting with "Error: ..." (e.g., "Error: Run not found").
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | Yes | Exact run_id (file stem, no extension). | |
| include_passed | No | If true, include passed tests in the output. Default false to keep context small. | |
| response_format | No | 'markdown' for human-readable, 'json' for programmatic. | markdown |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnly, idempotent, and non-destructive behavior. The description adds details about error responses and default test inclusion, providing useful extra context beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with three short paragraphs: purpose, parameter guidance, and return/error info. No unnecessary words, well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simple retrieval nature, annotations, and full schema coverage, the description provides all necessary context including error response format and reference to output schema. Complete for the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds value by explaining the default behavior of include_passed and the return format options, which goes beyond the schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Return a single test run by id' with a specific verb and resource. It is easily distinguishable from sibling tools qa_list_runs and qa_compare_runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains when to use include_passed and response_format parameters, but does not explicitly contrast with siblings. It provides clear context on default behaviors, which is helpful.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List available test runs from the configured runs directory.
Returns metadata only (run_id, suite, timestamps, counts) — not the full
list of test cases. Use `qa_get_run` for that.
Filters are applied in this order: suite (exact match), since, until.
Results are sorted by `started_at` ascending. Pagination via limit/offset.
Returns:
Markdown table or JSON depending on response_format. JSON shape:
{
"total": int,
"count": int,
"offset": int,
"has_more": bool,
"next_offset": int | null,
"items": [
{"run_id": str, "suite": str, "started_at": iso8601,
"summary": {"total","passed","failed","skipped","errors"}}
]
}
Error response: string starting with "Error:".
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max runs to return. | |
| since | No | Inclusive lower bound on started_at (ISO 8601, e.g. '2026-05-20T00:00:00Z'). | |
| suite | No | Filter by suite name (exact match). Omit to include all suites. | |
| until | No | Inclusive upper bound on started_at (ISO 8601). | |
| offset | No | Number of matching runs to skip. | |
| response_format | No | 'markdown' for human-readable, 'json' for programmatic. | markdown |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, destructiveHint=false, and idempotentHint=true, so the safety profile is clear. The description adds valuable behavioral context: returns metadata only, filter application order, sorting by started_at ascending, pagination via limit/offset, and return format options. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise at around 200 words, well-structured into logical paragraphs: purpose, clarification, filter order, return format, error handling. Every sentence serves a purpose with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the presence of an output schema (JSON shape provided), the description covers input, behavior, and output completely. It distinguishes from siblings, specifies error responses, and provides all necessary context for an agent to use the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema documents all parameters. The description adds context about filter order (suite, since, until) and sorting, but does not significantly enhance meaning beyond what schema descriptions already provide. Baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'List' and resource 'available test runs', and specifies it returns metadata only. It explicitly distinguishes itself from the sibling tool `qa_get_run`, which provides full test case details. The purpose is unambiguous and well-scoped.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit when-to-use guidance: for listing run metadata, and directs users to `qa_get_run` for full test cases. It does not explicitly mention when not to use compared to `qa_compare_runs`, but that sibling has a distinct purpose, so the guidance is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The three tools have clearly distinct purposes: listing runs, getting a single run's details, and comparing two runs. There is no overlap or ambiguity.
All tool names follow the consistent pattern 'qa_<verb>_runs' (list_runs, get_run, compare_runs), using snake_case and a clear verb_noun structure.
Three tools is on the lower end but still reasonable for a focused test run analysis toolkit. The count matches the scope of listing, retrieving, and comparing runs.
The set covers the core operations of browsing and comparing test runs, but it lacks tools for creating, updating, or deleting runs, and flakiness detection is explicitly omitted. There are notable gaps for full lifecycle management.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Conformance checker for MCP servers. Free, no key, verdicts recomputable and re-measured daily.
An MCP server that automatically collects feedback on your MCP server.
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server for querying and analyzing data from ad platforms, analytics tools, and spreadsheets
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/gabriel-tbc/qa-toolkit-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
