Skip to main content
Glama
test-intelligence-mcp

test-intelligence-mcp

test-intelligence-mcp

An MCP (Model Context Protocol) server that gives AI coding agents — Claude Code, Claude Desktop, or any other MCP client — the ability to analyse a Python repository's test health: coverage, flaky tests, and ML-based pull-request risk prediction.

Status: under active development. This README grows with each milestone; see Build status below for what's real today vs. what's still coming.

What it does

Point it at a Python repository and, from a natural-language conversation with an MCP-aware agent, you can:

  • Run the repo's test suite with coverage and get real, per-file numbers back (analyze_coverage)

  • Run the suite multiple times and detect genuinely flaky tests, as opposed to order-dependent or environment-dependent failures (detect_flaky_tests)

  • Persist test run results to Postgres to build a history over time (record_test_run)

  • Query that history back out (get_test_history)

  • Train a gradient-boosted classifier on accumulated run history to predict which files in a pull request are likely to break tests (train_risk_model)

  • Diff a branch against a base ref and get a ranked risk score per changed file (predict_pr_risk)

Everything is backed by real subprocess test execution and real parsing of coverage.json — nothing here scrapes terminal output or fakes numbers.

Related MCP server: Chisel

Why MCP

MCP is a protocol (open-sourced by Anthropic, now broadly adopted) that lets an AI agent discover and call tools exposed by a separate server process, over a standard JSON-RPC transport (stdio locally, or HTTP/SSE remotely). Instead of hand-rolling a custom API and teaching an agent's system prompt about it, you expose typed Python functions as "tools"; the client discovers their names, argument schemas, and docstrings automatically and calls them mid-conversation. This project uses FastMCP, the ergonomic Python SDK built on top of the official MCP spec — @mcp.tool() on a normal typed function is enough to expose it.

Tech stack

Concern

Choice

MCP server

FastMCP

Test execution

pytest, pytest-cov, coverage.py (parses coverage.json)

Database

PostgreSQL, async SQLAlchemy 2.0 (AsyncSession), asyncpg driver

Migrations

Alembic (versioned, no create_all())

ML

scikit-learn GradientBoostingClassifier

Git operations

GitPython / subprocess

CI

GitHub Actions

Local Postgres

Docker + docker-compose

Package management

uv

Repo layout

test-intelligence-mcp/
  src/test_intelligence/
    server.py       # FastMCP server + tool registration
    config.py        # typed settings, loaded from .env
    safety.py         # repo-path allowlist gate (see Safety below)
    paths.py            # cross-platform file-path normalization
    runners/               # pytest/coverage execution, JUnit + coverage.json parsing
    flaky/                   # multi-run comparison logic, order/seed control
    ml/                        # features.py, synthetic.py, training.py, prediction.py, model_store.py
    db/                        # SQLAlchemy models, session, query helpers
    git/                        # commit/branch metadata (repo_info.py), diff stats (diff.py)
  tests/                       # tests for THIS project's own code
    fixtures/                  # tiny throwaway repos the runner tests execute for real
  scripts/
    ci_report.py         # flaky-check + coverage summary, invoked by CI (see below)
  alembic/             # migration scripts
  .github/workflows/
    ci.yml              # runs on every PR — see Continuous Integration below
  docker-compose.yml  # local Postgres
  pyproject.toml
  .env.example

Setup

1. Prerequisites

  • Python 3.11+

  • uv — a fast, modern replacement for pip + venv + virtualenv, used here for dependency management and running commands. On Windows: winget install -e --id astral-sh.uv.

  • Docker Desktop — used to run Postgres locally via docker-compose, so you don't need Postgres installed on your machine. On Windows: winget install -e --id Docker.DockerDesktop.

2. Install dependencies

uv sync

uv sync reads pyproject.toml, resolves a locked dependency set (writing/using uv.lock), and creates a .venv/ — the uv equivalent of pip install -r requirements.txt inside a fresh virtualenv, but faster and reproducible across machines.

3. Start Postgres

docker-compose up -d

This starts a Postgres 16 container defined in docker-compose.yml, exposed on localhost:5433 with the credentials baked into that file. (Port 5433, not the Postgres-default 5432, to sidestep a collision if you already have Postgres installed natively — see docker-compose.yml for details.) -d runs it detached (in the background). Check it's healthy with:

docker-compose ps

You should see test-intelligence-postgres with status healthy.

4. Configure environment

cp .env.example .env

The defaults in .env.example already match docker-compose.yml's credentials, so for local dev you typically don't need to change anything except TI_ALLOWED_REPO_ROOTS (see Safety below).

5. Apply database migrations

uv run alembic upgrade head

Alembic replays every migration script under alembic/versions/ in order, bringing the database schema up to the latest version. Unlike SQLAlchemy's Base.metadata.create_all() (which can only stamp out tables matching whatever the current model code says, with no memory of past states), Alembic tracks schema history as an ordered chain of scripts — so changes are reviewable in git, reversible (alembic downgrade), and applied identically in dev, CI, and prod.

6. Register with an MCP client

Claude Code

claude mcp add test-intelligence -- uv run --directory "C:\path\to\test-intelligence-mcp" test-intelligence-mcp

Using --directory (rather than relying on whatever directory you happened to run claude mcp add from) makes the registration work regardless of where Claude Code's process actually launches the server from later — important because the server reads .env relative to its working directory at startup.

This registers the server as a stdio-transport MCP server scoped to your local Claude Code config. Verify it's connected with claude mcp list, then start a new Claude Code session (a session already running won't pick up a server registered after it started) and ask it to list available tools.

Claude Desktop

Add to claude_desktop_config.json (Windows: %APPDATA%\Claude\claude_desktop_config.json):

{
  "mcpServers": {
    "test-intelligence": {
      "command": "uv",
      "args": ["--directory", "C:\\path\\to\\test-intelligence-mcp", "run", "test-intelligence-mcp"]
    }
  }
}

Restart Claude Desktop; the six tools should appear under the 🔨 tools icon.

Safety

Because these tools execute a target repository's real test suite — i.e. arbitrary Python code — as a subprocess, two guardrails are enforced unconditionally:

  • Path allowlist: every repo_path argument is resolved to an absolute path and checked against TI_ALLOWED_REPO_ROOTS (a comma-separated list of permitted base directories) in .env. Paths outside the allowlist are rejected before any subprocess runs.

  • Subprocess timeouts: every subprocess call (a pytest run, a git command) has a hard timeout (TI_SUBPROCESS_TIMEOUT_SECONDS in .env, default 300s) so a hung or infinite-looping suite can't block the server indefinitely.

Usage examples

analyze_coverage

Ask an MCP-aware agent something like: "Run analyze_coverage on C:\path\to\some-repo". The tool runs that repo's test suite with coverage (using its own .venv/venv if it has one, otherwise falling back to this server's interpreter) and returns:

{
  "status": "ok",
  "tests_passed": true,
  "overall_coverage_percent": 87.5,
  "total_statements": 120,
  "total_covered_lines": 105,
  "total_uncovered_lines": 15,
  "files": [
    {
      "file": "pkg/calculator.py",
      "coverage_percent": 80.0,
      "num_statements": 10,
      "covered_lines": 8,
      "uncovered_line_count": 2,
      "uncovered_lines": [12, 13]
    }
  ]
}

files is sorted worst-covered first, so an agent can immediately point at the files most in need of tests. Requires the target repo to have pytest and pytest-cov installed in whatever Python environment gets resolved.

record_test_run + get_test_history

"Record a test run for C:\path\to\some-repo, then show me the history for pkg/calculator.py" — the first call executes the suite once via pytest's --junitxml output (so pytest alone is enough in the target repo, no plugin needed), persists a Repository/TestRun/TestResult row set to Postgres, and tags the run with the target repo's current commit SHA and branch (via GitPython) when it's a real git repo:

{
  "status": "ok",
  "run_id": 3,
  "repo_id": 1,
  "commit_sha": "a1b2c3d...",
  "branch": "main",
  "duration_seconds": 0.52,
  "total_tests": 3,
  "passed_count": 1,
  "failed_count": 1,
  "skipped_count": 1
}

get_test_history only ever reads rows already written by record_test_run — it never triggers a run itself, and it queries across every repo this server has ever recorded (there's no repo_path argument), optionally filtered to one file_path:

{
  "status": "ok",
  "count": 2,
  "history": [
    {
      "repo_name": "C:\\path\\to\\some-repo",
      "run_id": 3,
      "commit_sha": "a1b2c3d...",
      "branch": "main",
      "started_at": "2026-08-16T00:20:11+00:00",
      "node_id": "tests/test_calculator.py::test_divide",
      "file_path": "tests/test_calculator.py",
      "outcome": "passed",
      "duration_seconds": 0.001,
      "error_message": null
    }
  ]
}

detect_flaky_tests

"Run detect_flaky_tests on C:\path\to\some-repo with 5 runs" — runs the suite runs times with known test-order-randomizing plugins (pytest-randomly, pytest-random-order) explicitly disabled, so test order is identical every run. That isolates genuine non-determinism (timing, shared state, unseeded randomness in the code under test) as the only possible explanation for a test disagreeing with itself across runs — an order-shuffling plugin would otherwise make order-dependent failures indistinguishable from real flakiness. Progress streams live via MCP progress notifications (visible to clients that support them) since 5+ sequential runs on a large suite can take a while:

{
  "status": "ok",
  "repo_id": 2,
  "runs_requested": 5,
  "runs_completed": 5,
  "total_tests_observed": 2,
  "flaky_test_count": 1,
  "flaky_tests": [
    {
      "node_id": "tests/test_flaky.py::test_alternates",
      "runs_observed": 5,
      "inconsistency_count": 2,
      "flakiness_rate": 0.4,
      "outcomes": ["passed", "failed", "passed", "failed", "passed"],
      "majority_outcome": "passed"
    }
  ],
  "run_failures": []
}

Detected flaky tests are also persisted to the flaky_reports table.

train_risk_model

"Train the risk model" — trains a GradientBoostingClassifier to predict "will a test fail after this file changes?" from 10 features per file (churn, historical failure count, current coverage, test count touching the file, days since last modified, distinct authors, file size, cyclomatic complexity — see Cold-start strategy for where the training data comes from), evaluates on a held-out split, and reports honestly:

{
  "status": "ok",
  "model_path": "models/risk_model.joblib",
  "real_sample_count": 0,
  "synthetic_sample_count": 500,
  "total_sample_count": 500,
  "test_set_size": 125,
  "metrics": {
    "accuracy": 0.6,
    "precision": 0.5962,
    "recall": 0.5167,
    "f1": 0.5536
  },
  "caveat": "Only 0 real training example(s) recorded so far (via record_test_run) — this training run is dominated by synthetic, artificially-generated bootstrap data. These metrics describe how well the model fits that synthetic relationship, NOT real predictive power on an actual repository. Keep calling record_test_run on real repos, then retrain, before trusting these numbers for anything beyond confirming the training pipeline itself works."
}

The caveat field only disappears once real_sample_count clears a real threshold (30, see ml/training.py) — this tool never presents synthetic-dominated metrics as if they were validated against reality.

(Remaining tools filled in per-milestone — see Build status.)

Cold-start strategy for the ML model

train_risk_model needs labelled examples — "given these features about a file change, did a test tied to that file fail afterward?" On a freshly set up server, zero runs have been recorded, so there's no history to learn from. Three options were weighed before writing any ML code:

  1. Replay a real open-source repo's git/CI history. Clone a real project, walk its commits, check each one out, install its dependencies as they existed at that point in history, run its suite, extract real features and real labels. The most realistic data by far — but expensive and fragile to build reliably: dependency installation breaks across years of history (deprecated packages, Python version drift), full-history checkouts are slow, and it bolts a hard external dependency (a specific repo, at a specific point in time) onto this project's own CI, which would need to reproduce it on every run.

  2. Replay this project's own commits. Same idea, smaller scope — doesn't avoid the core cost problem, and this project's own history is far too short and narrow to represent the breadth of file-change patterns a general-purpose risk model should generalize across.

  3. Generate synthetic data (chosen). Draw feature vectors from plausible distributions and derive labels from a deliberately-designed, domain-informed generative rule — more churn + more past failures + lower coverage + higher complexity → higher failure probability, plus noise — rather than a coin flip. Fast, fully reproducible, no external repo required, and enough to exercise the entire pipeline (feature extraction → training → evaluation) honestly today.

The honest tradeoff: a model trained purely on synthetic data has learned the shape of a plausible risk relationship, not the real one. Its metrics on held-out synthetic data look reasonable (~0.6 accuracy, ROC-AUC ~0.67 — see tests/ml/test_synthetic.py), which only proves the pipeline works, not that it predicts anything about a real repository. train_risk_model blends in real examples the moment any exist (via record_test_run → file_changes rows — see below) and always reports the real/synthetic split plus an explicit caveat when real data is too thin to trust, rather than ever presenting synthetic-derived numbers as validated.

Where real examples come from: record_test_run computes a real git diff (HEAD~1..HEAD) after every run and writes one file_changes row per changed file, labelled tests_failed_after = "did any test fail in this run" — applied to every file changed in it, not per-file attribution. That's a deliberate choice: attributing a failure to the specific file that caused it would need coverage-based tracing (which test executed which source lines), which this project doesn't do. The coarser signal is honestly correlational ("this file was part of a commit that broke something"), not causal — see the comment in runners/record_run.py for the full reasoning, including why a file-path-string-match heuristic would have looked more precise while actually being narrower and more misleading.

Historical feature extraction (used for real training examples) reads git history "as of" each recorded run's own timestamp and commit — git log --before, git show <sha>:<path> — never the file's current state, so a model can't accidentally train on information that didn't exist yet at prediction time. coverage_percent is the one feature that can't be reconstructed historically without re-running the full suite at that exact commit (too expensive to do per training example), so it's stored as an explicit "unknown" sentinel for real historical rows, and only computed fresh for live predictions (predict_pr_risk, below).

predict_pr_risk

"Predict PR risk for C:\path\to\some-repo against main" — diffs base_ref..HEAD (a real git diff --numstat), extracts live features for each changed file (current working-tree state, plus a fresh analyze_coverage run for real current coverage — not the "unknown" sentinel historical training rows get), scores each with the trained model, and ranks highest-risk first:

{
  "status": "ok",
  "repo_id": 3,
  "base_ref": "0bcbeba860df0457c55ad3c1d3826ed5fd941506",
  "commit_sha": "8306f4598275f92907de89e1161f982772f3aac7",
  "model_trained_at": "2026-08-17T19:03:42.707577+00:00",
  "model_real_sample_count": 0,
  "predictions": [
    { "file": "tests/test_calculator.py", "predicted_risk_probability": 0.0743, "lines_added": 9, "lines_deleted": 1 },
    { "file": "pkg/calculator.py", "predicted_risk_probability": 0.0457, "lines_added": 7, "lines_deleted": 0 }
  ]
}

model_real_sample_count is carried through from the model's training metadata — so a caller can see at a glance whether these predictions come from a synthetic-dominated model (see Cold-start strategy) without a separate lookup. Requires train_risk_model to have been run at least once (no_trained_model error otherwise) — this tool never trains a model implicitly as a side effect. Every prediction is persisted to risk_predictions with actual_outcome left NULL, so a real repo's predictions can eventually be checked against what actually happened — that evaluation isn't built yet, but the data is captured from day one so it can be added without a schema change.

Continuous Integration (GitHub Actions)

GitHub Actions is CI built directly into GitHub: a workflow — one YAML file, .github/workflows/ci.yml — describes jobs that run automatically in response to repo events (here: opening/updating a pull request, or pushing to main). Each job runs on a fresh, throwaway virtual machine (a "runner") — nothing persists between runs except what's explicitly cached or uploaded — and is just a sequence of steps, each either a shell command or a reusable action (a packaged step someone else published, referenced like actions/checkout@v4).

This project's workflow is the project testing itself, end to end:

  1. Check out the repo and install uv + dependencies — same tools a contributor installs locally.

  2. Start Postgres as a service container — a second container GitHub Actions runs alongside the job, reachable at localhost:5433 from every step, exactly like docker-compose up -d locally but managed by GitHub instead of Docker Desktop. The job's steps don't start until its healthcheck passes — no hand-written "wait for Postgres" polling loop needed.

  3. Apply Alembic migrations, then run this project's own test suite with --cov-fail-under=$COVERAGE_THRESHOLD — pytest-cov's built-in gate; the build fails outright if coverage drops below it (currently 80%, with headroom under the real ~93%).

  4. Run detect_flaky_tests against this project's own tests — via scripts/ci_report.py, which calls the tool through the real MCP layer (fastmcp.Client talking to the actual server object), not a shortcut. Informational only — it never fails the build, only coverage does.

  5. Upload both reports as workflow artifacts (actions/upload-artifact) — downloadable from the workflow run's page for 90 days by default.

  6. Write a job summary ($GITHUB_STEP_SUMMARY, rendered as Markdown directly on the run's page) and post it as a PR comment (actions/github-script, using the run's built-in GITHUB_TOKEN — no extra secrets needed). The step summary is the fallback that always works, including for PRs from forks, which get a read-only token that can't post comments (a GitHub security restriction, not a bug in this workflow) — the comment step is wrapped in continue-on-error: true so that limitation degrades gracefully instead of failing the whole job.

To actually see this run, the project needs to live in a real GitHub repository with commits pushed to it — nothing in this local build process has created one yet. Once that exists: open a PR, and the Actions tab (and the PR itself, once the comment lands) shows it running live.

Build status

This project is built milestone-by-milestone, each one verified working before moving to the next.

  • Milestone 1 — project skeleton, docker-compose Postgres, .env.example, this README

  • Milestone 2 — database layer (SQLAlchemy models + Alembic)

  • Milestone 3 — FastMCP server skeleton (6 tools registered, placeholder bodies)

  • Milestone 4 — analyze_coverage

  • Milestone 5 — record_test_run + get_test_history

  • Milestone 6 — detect_flaky_tests

  • Milestone 7 — ML cold-start strategy, feature extraction, train_risk_model

  • Milestone 8 — predict_pr_risk

  • Milestone 9 — GitHub Actions CI (built + locally verified; live PR run pending a real GitHub repo)

  • Milestone 10 — final polish

Running this project's own tests

uv sync --extra dev     # installs pytest-asyncio + ruff on top of the base deps
uv run pytest -v
uv run pytest --cov --cov-report=term-missing   # with coverage
uv run ruff check .                              # lint

The DB-layer tests use a real, disposable Postgres database (test_intelligence_test, created and torn down automatically) and run the actual Alembic migrations against it, rather than mocking the database or using create_all() — the same approach CI uses via its Postgres service container (Milestone 9). See tests/conftest.py for details.

Data model

Six tables, managed by Alembic migrations:

  • repositories — a tracked repo (name + local path or remote URL)

  • test_runs — one row per pytest invocation (repo, commit SHA, branch, timestamp, duration, pass/fail/skip counts)

  • test_results — one row per test node ID within a run (outcome, duration, error message)

  • file_changes — per-file diff stats for a run (lines added/deleted, whether tests failed after the change)

  • flaky_reports — per-test-node flakiness summary (runs observed, inconsistency count, detection timestamp)

  • risk_predictions — per-file ML risk scores for a commit, plus actual outcome once known (for offline evaluation of the model)

License

MIT

Available Tools

6 tools
analyze_coverageA

Run the test suite with coverage and report real, per-file coverage numbers.

Executes pytest with pytest-cov against the target repository, then parses the machine-readable coverage.json report it produces (never the human-readable terminal summary) to compute overall coverage percentage, per-file coverage, and uncovered line counts.

Inputs: repo_path (required), test_path. Returns: object

detect_flaky_testsA

Run the test suite multiple times and flag tests with inconsistent outcomes.

Runs the full suite runs times, comparing each test's outcome (by its stable pytest node ID) across runs. A test that passes in some runs and fails in others is reported as flaky, along with its flakiness rate. Test order and random seeds are controlled so genuine flakiness isn't confused with order-dependent or environment-dependent failures.

Inputs: repo_path (required), runs. Returns: object

get_test_historyA

Query previously recorded test run history from the database.

Reads rows written by record_test_run — this tool never triggers a new test run itself, it only reports on what's already been recorded.

Inputs: limit, file_path. Returns: object

predict_pr_riskA

Diff the current branch against a base ref and rank changed files by predicted risk.

Computes the file-level diff between HEAD and base_ref, extracts the same features the risk model was trained on for each changed file (churn, historical failure count, current coverage, etc.), and runs them through the trained GradientBoostingClassifier to produce a ranked risk score per file.

Inputs: repo_path (required), base_ref. Returns: object

record_test_runA

Execute the test suite once and persist the results to the database.

This is the explicit, distinct step that builds up the training data other tools (get_test_history, train_risk_model) rely on — it never runs implicitly as a side effect of another tool, so it's always clear when history is being written versus just read or analysed.

Inputs: repo_path (required). Returns: object

train_risk_modelA

Retrain the PR-risk classifier on everything currently in the database.

Trains a GradientBoostingClassifier using file_changes/test_results history as labelled examples (did a test tied to this file fail within N runs after the file changed?), then reports accuracy, precision, recall, and F1 on a held-out split. When the training set is small, the returned metrics come with an explicit caveat that they aren't yet statistically meaningful — this tool never reports metrics as trustworthy without that context.

Returns: Training set size and accuracy/precision/recall/F1, with an honest caveat attached when the training set is too small for the metrics to mean much.

Inputs: no named parameters. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.2/5.0

    Scored across 6 tools

    Disambiguation5/5

    Each tool targets a distinct function: coverage analysis, flaky detection, risk prediction, test execution recording, history querying, and model training. There is no functional overlap, and the descriptions clearly differentiate their purposes.

    Naming Consistency4/5

    Tool names consistently use a verb_noun pattern (e.g., analyze_coverage, detect_flaky_tests, train_risk_model). The pattern is clear and predictable, though 'record_test_run' and 'get_test_history' slightly shift the pattern (record vs. get, test_run vs. test_history), but it's still minor and readable.

    Tool Count5/5

    With 6 tools, the set is well-scoped for a CI/testing intelligence server. Each tool covers a necessary stage (record, query, analyze, detect flakiness, predict risk, train model), and none feel redundant or extraneous.

    Completeness4/5

    The tool set covers the core lifecycle: recording test runs (record_test_run), querying history (get_test_history), analyzing coverage (analyze_coverage), detecting flakiness (detect_flaky_tests), predicting risk (predict_pr_risk), and training the risk model (train_risk_model). A minor gap is the lack of a tool to delete or manage stored history, but the essential pipeline is complete.

    Maintenance

    ActivityInactive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      A production-ready MCP server that provides comprehensive dbt project quality assessment for any GitHub repository, enabling AI agents to analyze dbt models, check metadata coverage, and map data lineage.
      8
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      MCP server for test impact analysis and code intelligence. Maps tests to code and git history to determine impacted tests, risk scores, and ownership for AI coding agents.
      8
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      This MCP server enables automated maintenance and code analysis for Python/pytest repositories in isolated Docker environments. It supports read-only investigations, fix-and-verify tasks, and provides full audit trails with SQLite event history and artifact exports.
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Analyzes codebases to produce structured quality health reports with risk-scored modules based on git churn, test coverage, and test-to-source mapping, acting as an MCP server for AI coding agents and a standalone CLI.
      MIT