# test-intelligence-mcp

An [MCP](https://modelcontextprotocol.io) (Model Context Protocol) server that gives AI coding
agents — Claude Code, Claude Desktop, or any other MCP client — the ability to analyse a Python
repository's test health: coverage, flaky tests, and ML-based pull-request risk prediction.

> **Status:** under active development. This README grows with each milestone; see
> [Build status](#build-status) below for what's real today vs. what's still coming.

## What it does

Point it at a Python repository and, from a natural-language conversation with an MCP-aware
agent, you can:

- Run the repo's test suite with coverage and get real, per-file numbers back (`analyze_coverage`)
- Run the suite multiple times and detect genuinely flaky tests, as opposed to
  order-dependent or environment-dependent failures (`detect_flaky_tests`)
- Persist test run results to Postgres to build a history over time (`record_test_run`)
- Query that history back out (`get_test_history`)
- Train a gradient-boosted classifier on accumulated run history to predict which files in a
  pull request are likely to break tests (`train_risk_model`)
- Diff a branch against a base ref and get a ranked risk score per changed file
  (`predict_pr_risk`)

Everything is backed by real subprocess test execution and real parsing of `coverage.json` —
nothing here scrapes terminal output or fakes numbers.

## Why MCP

MCP is a protocol (open-sourced by Anthropic, now broadly adopted) that lets an AI agent
discover and call tools exposed by a separate server process, over a standard JSON-RPC
transport (stdio locally, or HTTP/SSE remotely). Instead of hand-rolling a custom API and
teaching an agent's system prompt about it, you expose typed Python functions as "tools"; the
client discovers their names, argument schemas, and docstrings automatically and calls them
mid-conversation. This project uses **[FastMCP](https://github.com/jlowin/fastmcp)**, the
ergonomic Python SDK built on top of the official MCP spec — `@mcp.tool()` on a normal typed
function is enough to expose it.

## Tech stack

| Concern | Choice |
|---|---|
| MCP server | [FastMCP](https://github.com/jlowin/fastmcp) |
| Test execution | pytest, pytest-cov, coverage.py (parses `coverage.json`) |
| Database | PostgreSQL, async SQLAlchemy 2.0 (`AsyncSession`), asyncpg driver |
| Migrations | Alembic (versioned, no `create_all()`) |
| ML | scikit-learn `GradientBoostingClassifier` |
| Git operations | GitPython / subprocess |
| CI | GitHub Actions |
| Local Postgres | Docker + docker-compose |
| Package management | [uv](https://docs.astral.sh/uv/) |

## Repo layout

```
test-intelligence-mcp/
  src/test_intelligence/
    server.py       # FastMCP server + tool registration
    config.py        # typed settings, loaded from .env
    safety.py         # repo-path allowlist gate (see Safety below)
    paths.py            # cross-platform file-path normalization
    runners/               # pytest/coverage execution, JUnit + coverage.json parsing
    flaky/                   # multi-run comparison logic, order/seed control
    ml/                        # features.py, synthetic.py, training.py, prediction.py, model_store.py
    db/                        # SQLAlchemy models, session, query helpers
    git/                        # commit/branch metadata (repo_info.py), diff stats (diff.py)
  tests/                       # tests for THIS project's own code
    fixtures/                  # tiny throwaway repos the runner tests execute for real
  scripts/
    ci_report.py         # flaky-check + coverage summary, invoked by CI (see below)
  alembic/             # migration scripts
  .github/workflows/
    ci.yml              # runs on every PR — see Continuous Integration below
  docker-compose.yml  # local Postgres
  pyproject.toml
  .env.example
```

## Setup

### 1. Prerequisites

- Python 3.11+
- [uv](https://docs.astral.sh/uv/getting-started/installation/) — a fast, modern replacement
  for pip + venv + virtualenv, used here for dependency management and running commands. On
  Windows: `winget install -e --id astral-sh.uv`.
- [Docker Desktop](https://www.docker.com/products/docker-desktop/) — used to run Postgres
  locally via docker-compose, so you don't need Postgres installed on your machine. On
  Windows: `winget install -e --id Docker.DockerDesktop`.

### 2. Install dependencies

```bash
uv sync
```

`uv sync` reads `pyproject.toml`, resolves a locked dependency set (writing/using
`uv.lock`), and creates a `.venv/` — the uv equivalent of `pip install -r requirements.txt`
inside a fresh virtualenv, but faster and reproducible across machines.

### 3. Start Postgres

```bash
docker-compose up -d
```

This starts a Postgres 16 container defined in `docker-compose.yml`, exposed on
`localhost:5433` with the credentials baked into that file. (Port 5433, not the
Postgres-default 5432, to sidestep a collision if you already have Postgres installed
natively — see `docker-compose.yml` for details.) `-d` runs it detached (in the
background). Check it's healthy with:

```bash
docker-compose ps
```

You should see `test-intelligence-postgres` with status `healthy`.

### 4. Configure environment

```bash
cp .env.example .env
```

The defaults in `.env.example` already match `docker-compose.yml`'s credentials, so for local
dev you typically don't need to change anything except `TI_ALLOWED_REPO_ROOTS` (see
[Safety](#safety) below).

### 5. Apply database migrations

```bash
uv run alembic upgrade head
```

Alembic replays every migration script under `alembic/versions/` in order, bringing the
database schema up to the latest version. Unlike SQLAlchemy's `Base.metadata.create_all()`
(which can only stamp out tables matching whatever the current model code says, with no
memory of past states), Alembic tracks schema history as an ordered chain of scripts — so
changes are reviewable in git, reversible (`alembic downgrade`), and applied identically in
dev, CI, and prod.

### 6. Register with an MCP client

#### Claude Code

```bash
claude mcp add test-intelligence -- uv run --directory "C:\path\to\test-intelligence-mcp" test-intelligence-mcp
```

Using `--directory` (rather than relying on whatever directory you happened to run
`claude mcp add` from) makes the registration work regardless of where Claude Code's
process actually launches the server from later — important because the server reads
`.env` relative to its working directory at startup.

This registers the server as a stdio-transport MCP server scoped to your local Claude Code
config. Verify it's connected with `claude mcp list`, then start a **new** Claude Code
session (a session already running won't pick up a server registered after it started)
and ask it to list available tools.

#### Claude Desktop

Add to `claude_desktop_config.json` (Windows:
`%APPDATA%\Claude\claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "test-intelligence": {
      "command": "uv",
      "args": ["--directory", "C:\\path\\to\\test-intelligence-mcp", "run", "test-intelligence-mcp"]
    }
  }
}
```

Restart Claude Desktop; the six tools should appear under the 🔨 tools icon.

## Safety

Because these tools execute a target repository's real test suite — i.e. arbitrary Python
code — as a subprocess, two guardrails are enforced unconditionally:

- **Path allowlist**: every `repo_path` argument is resolved to an absolute path and checked
  against `TI_ALLOWED_REPO_ROOTS` (a comma-separated list of permitted base directories) in
  `.env`. Paths outside the allowlist are rejected before any subprocess runs.
- **Subprocess timeouts**: every `subprocess` call (a pytest run, a git command) has a hard
  timeout (`TI_SUBPROCESS_TIMEOUT_SECONDS` in `.env`, default 300s) so a hung or
  infinite-looping suite can't block the server indefinitely.

## Usage examples

### `analyze_coverage`

Ask an MCP-aware agent something like: *"Run analyze_coverage on
C:\path\to\some-repo"*. The tool runs that repo's test suite with coverage (using its
own `.venv`/`venv` if it has one, otherwise falling back to this server's interpreter)
and returns:

```json
{
  "status": "ok",
  "tests_passed": true,
  "overall_coverage_percent": 87.5,
  "total_statements": 120,
  "total_covered_lines": 105,
  "total_uncovered_lines": 15,
  "files": [
    {
      "file": "pkg/calculator.py",
      "coverage_percent": 80.0,
      "num_statements": 10,
      "covered_lines": 8,
      "uncovered_line_count": 2,
      "uncovered_lines": [12, 13]
    }
  ]
}
```

`files` is sorted worst-covered first, so an agent can immediately point at the files
most in need of tests. Requires the target repo to have `pytest` and `pytest-cov`
installed in whatever Python environment gets resolved.

### `record_test_run` + `get_test_history`

*"Record a test run for C:\path\to\some-repo, then show me the history for
pkg/calculator.py"* — the first call executes the suite once via pytest's
`--junitxml` output (so `pytest` alone is enough in the target repo, no plugin
needed), persists a `Repository`/`TestRun`/`TestResult` row set to Postgres, and tags
the run with the target repo's current commit SHA and branch (via GitPython) when
it's a real git repo:

```json
{
  "status": "ok",
  "run_id": 3,
  "repo_id": 1,
  "commit_sha": "a1b2c3d...",
  "branch": "main",
  "duration_seconds": 0.52,
  "total_tests": 3,
  "passed_count": 1,
  "failed_count": 1,
  "skipped_count": 1
}
```

`get_test_history` only ever reads rows already written by `record_test_run` — it
never triggers a run itself, and it queries across every repo this server has ever
recorded (there's no `repo_path` argument), optionally filtered to one `file_path`:

```json
{
  "status": "ok",
  "count": 2,
  "history": [
    {
      "repo_name": "C:\\path\\to\\some-repo",
      "run_id": 3,
      "commit_sha": "a1b2c3d...",
      "branch": "main",
      "started_at": "2026-08-16T00:20:11+00:00",
      "node_id": "tests/test_calculator.py::test_divide",
      "file_path": "tests/test_calculator.py",
      "outcome": "passed",
      "duration_seconds": 0.001,
      "error_message": null
    }
  ]
}
```

### `detect_flaky_tests`

*"Run detect_flaky_tests on C:\path\to\some-repo with 5 runs"* — runs the suite `runs`
times with known test-order-randomizing plugins (`pytest-randomly`, `pytest-random-order`)
explicitly disabled, so test order is identical every run. That isolates genuine
non-determinism (timing, shared state, unseeded randomness in the code under test)
as the only possible explanation for a test disagreeing with itself across runs —
an order-shuffling plugin would otherwise make order-dependent failures
indistinguishable from real flakiness. Progress streams live via MCP progress
notifications (visible to clients that support them) since 5+ sequential runs on a
large suite can take a while:

```json
{
  "status": "ok",
  "repo_id": 2,
  "runs_requested": 5,
  "runs_completed": 5,
  "total_tests_observed": 2,
  "flaky_test_count": 1,
  "flaky_tests": [
    {
      "node_id": "tests/test_flaky.py::test_alternates",
      "runs_observed": 5,
      "inconsistency_count": 2,
      "flakiness_rate": 0.4,
      "outcomes": ["passed", "failed", "passed", "failed", "passed"],
      "majority_outcome": "passed"
    }
  ],
  "run_failures": []
}
```

Detected flaky tests are also persisted to the `flaky_reports` table.

### `train_risk_model`

*"Train the risk model"* — trains a `GradientBoostingClassifier` to predict "will a
test fail after this file changes?" from 10 features per file (churn, historical
failure count, current coverage, test count touching the file, days since last
modified, distinct authors, file size, cyclomatic complexity — see
[Cold-start strategy](#cold-start-strategy-for-the-ml-model) for where the training
data comes from), evaluates on a held-out split, and reports honestly:

```json
{
  "status": "ok",
  "model_path": "models/risk_model.joblib",
  "real_sample_count": 0,
  "synthetic_sample_count": 500,
  "total_sample_count": 500,
  "test_set_size": 125,
  "metrics": {
    "accuracy": 0.6,
    "precision": 0.5962,
    "recall": 0.5167,
    "f1": 0.5536
  },
  "caveat": "Only 0 real training example(s) recorded so far (via record_test_run) — this training run is dominated by synthetic, artificially-generated bootstrap data. These metrics describe how well the model fits that synthetic relationship, NOT real predictive power on an actual repository. Keep calling record_test_run on real repos, then retrain, before trusting these numbers for anything beyond confirming the training pipeline itself works."
}
```

The `caveat` field only disappears once `real_sample_count` clears a real threshold
(30, see `ml/training.py`) — this tool never presents synthetic-dominated metrics as
if they were validated against reality.

*(Remaining tools filled in per-milestone — see [Build status](#build-status).)*

## Cold-start strategy for the ML model

`train_risk_model` needs labelled examples — "given these features about a file
change, did a test tied to that file fail afterward?" On a freshly set up server,
zero runs have been recorded, so there's no history to learn from. Three options
were weighed before writing any ML code:

1. **Replay a real open-source repo's git/CI history.** Clone a real project, walk
   its commits, check each one out, install its dependencies as they existed at
   that point in history, run its suite, extract real features and real labels.
   The most *realistic* data by far — but expensive and fragile to build reliably:
   dependency installation breaks across years of history (deprecated packages,
   Python version drift), full-history checkouts are slow, and it bolts a hard
   external dependency (a specific repo, at a specific point in time) onto this
   project's own CI, which would need to reproduce it on every run.
2. **Replay this project's own commits.** Same idea, smaller scope — doesn't avoid
   the core cost problem, and this project's own history is far too short and
   narrow to represent the breadth of file-change patterns a general-purpose risk
   model should generalize across.
3. **Generate synthetic data (chosen).** Draw feature vectors from plausible
   distributions and derive labels from a deliberately-designed, domain-informed
   generative rule — more churn + more past failures + lower coverage + higher
   complexity → higher failure probability, plus noise — rather than a coin flip.
   Fast, fully reproducible, no external repo required, and enough to exercise the
   *entire* pipeline (feature extraction → training → evaluation) honestly today.

**The honest tradeoff:** a model trained purely on synthetic data has learned the
*shape* of a plausible risk relationship, not the real one. Its metrics on
held-out synthetic data look reasonable (~0.6 accuracy, ROC-AUC ~0.67 — see
`tests/ml/test_synthetic.py`), which only proves the pipeline works, not that it
predicts anything about a real repository. `train_risk_model` blends in real
examples the moment any exist (via `record_test_run` → `file_changes` rows — see
below) and always reports the real/synthetic split plus an explicit caveat when
real data is too thin to trust, rather than ever presenting synthetic-derived
numbers as validated.

**Where real examples come from:** `record_test_run` computes a real git diff
(`HEAD~1..HEAD`) after every run and writes one `file_changes` row per changed
file, labelled `tests_failed_after` = "did *any* test fail in this run" — applied
to every file changed in it, not per-file attribution. That's a deliberate choice:
attributing a failure to the *specific* file that caused it would need
coverage-based tracing (which test executed which source lines), which this
project doesn't do. The coarser signal is honestly correlational ("this file was
part of a commit that broke something"), not causal — see the comment in
`runners/record_run.py` for the full reasoning, including why a
file-path-string-match heuristic would have looked more precise while actually
being narrower and more misleading.

Historical feature extraction (used for real training examples) reads git history
"as of" each recorded run's own timestamp and commit — `git log --before`, `git
show <sha>:<path>` — never the file's *current* state, so a model can't
accidentally train on information that didn't exist yet at prediction time.
`coverage_percent` is the one feature that can't be reconstructed historically
without re-running the full suite at that exact commit (too expensive to do per
training example), so it's stored as an explicit "unknown" sentinel for real
historical rows, and only computed fresh for live predictions (`predict_pr_risk`,
below).

### `predict_pr_risk`

*"Predict PR risk for C:\path\to\some-repo against main"* — diffs `base_ref..HEAD`
(a real `git diff --numstat`), extracts *live* features for each changed file
(current working-tree state, plus a fresh `analyze_coverage` run for real current
coverage — not the "unknown" sentinel historical training rows get), scores each
with the trained model, and ranks highest-risk first:

```json
{
  "status": "ok",
  "repo_id": 3,
  "base_ref": "0bcbeba860df0457c55ad3c1d3826ed5fd941506",
  "commit_sha": "8306f4598275f92907de89e1161f982772f3aac7",
  "model_trained_at": "2026-08-17T19:03:42.707577+00:00",
  "model_real_sample_count": 0,
  "predictions": [
    { "file": "tests/test_calculator.py", "predicted_risk_probability": 0.0743, "lines_added": 9, "lines_deleted": 1 },
    { "file": "pkg/calculator.py", "predicted_risk_probability": 0.0457, "lines_added": 7, "lines_deleted": 0 }
  ]
}
```

`model_real_sample_count` is carried through from the model's training metadata —
so a caller can see at a glance whether these predictions come from a
synthetic-dominated model (see [Cold-start strategy](#cold-start-strategy-for-the-ml-model))
without a separate lookup. Requires `train_risk_model` to have been run at least
once (`no_trained_model` error otherwise) — this tool never trains a model
implicitly as a side effect. Every prediction is persisted to `risk_predictions`
with `actual_outcome` left `NULL`, so a real repo's predictions can eventually be
checked against what actually happened — that evaluation isn't built yet, but the
data is captured from day one so it can be added without a schema change.

## Continuous Integration (GitHub Actions)

GitHub Actions is CI built directly into GitHub: a **workflow** — one YAML file,
[.github/workflows/ci.yml](.github/workflows/ci.yml) — describes **jobs** that run
automatically in response to repo events (here: opening/updating a pull request, or
pushing to `main`). Each job runs on a fresh, throwaway virtual machine (a
"runner") — nothing persists between runs except what's explicitly cached or
uploaded — and is just a sequence of **steps**, each either a shell command or a
reusable **action** (a packaged step someone else published, referenced like
`actions/checkout@v4`).

This project's workflow is the project testing itself, end to end:

1. **Check out the repo** and **install `uv` + dependencies** — same tools a
   contributor installs locally.
2. **Start Postgres as a service container** — a second container GitHub Actions
   runs alongside the job, reachable at `localhost:5433` from every step, exactly
   like `docker-compose up -d` locally but managed by GitHub instead of Docker
   Desktop. The job's steps don't start until its healthcheck passes — no
   hand-written "wait for Postgres" polling loop needed.
3. **Apply Alembic migrations**, then **run this project's own test suite** with
   `--cov-fail-under=$COVERAGE_THRESHOLD` — pytest-cov's built-in gate; the build
   fails outright if coverage drops below it (currently 80%, with headroom under
   the real ~93%).
4. **Run `detect_flaky_tests` against this project's own tests** — via
   [scripts/ci_report.py](scripts/ci_report.py), which calls the tool through the
   *real* MCP layer (`fastmcp.Client` talking to the actual server object), not a
   shortcut. Informational only — it never fails the build, only coverage does.
5. **Upload both reports as workflow artifacts** (`actions/upload-artifact`) —
   downloadable from the workflow run's page for 90 days by default.
6. **Write a job summary** (`$GITHUB_STEP_SUMMARY`, rendered as Markdown directly
   on the run's page) **and post it as a PR comment** (`actions/github-script`,
   using the run's built-in `GITHUB_TOKEN` — no extra secrets needed). The step
   summary is the fallback that always works, including for PRs from forks, which
   get a read-only token that can't post comments (a GitHub security restriction,
   not a bug in this workflow) — the comment step is wrapped in
   `continue-on-error: true` so that limitation degrades gracefully instead of
   failing the whole job.

**To actually see this run**, the project needs to live in a real GitHub
repository with commits pushed to it — nothing in this local build process has
created one yet. Once that exists: open a PR, and the Actions tab (and the PR
itself, once the comment lands) shows it running live.

## Build status

This project is built milestone-by-milestone, each one verified working before moving to the
next.

- [x] **Milestone 1** — project skeleton, docker-compose Postgres, `.env.example`, this README
- [x] **Milestone 2** — database layer (SQLAlchemy models + Alembic)
- [x] **Milestone 3** — FastMCP server skeleton (6 tools registered, placeholder bodies)
- [x] **Milestone 4** — `analyze_coverage`
- [x] **Milestone 5** — `record_test_run` + `get_test_history`
- [x] **Milestone 6** — `detect_flaky_tests`
- [x] **Milestone 7** — ML cold-start strategy, feature extraction, `train_risk_model`
- [x] **Milestone 8** — `predict_pr_risk`
- [x] **Milestone 9** — GitHub Actions CI (built + locally verified; live PR run pending a real GitHub repo)
- [ ] **Milestone 10** — final polish

## Running this project's own tests

```bash
uv sync --extra dev     # installs pytest-asyncio + ruff on top of the base deps
uv run pytest -v
uv run pytest --cov --cov-report=term-missing   # with coverage
uv run ruff check .                              # lint
```

The DB-layer tests use a real, disposable Postgres database (`test_intelligence_test`,
created and torn down automatically) and run the actual Alembic migrations against it,
rather than mocking the database or using `create_all()` — the same approach CI uses
via its Postgres service container (Milestone 9). See `tests/conftest.py` for details.

## Data model

Six tables, managed by Alembic migrations:

- **repositories** — a tracked repo (name + local path or remote URL)
- **test_runs** — one row per `pytest` invocation (repo, commit SHA, branch, timestamp,
  duration, pass/fail/skip counts)
- **test_results** — one row per test node ID within a run (outcome, duration, error message)
- **file_changes** — per-file diff stats for a run (lines added/deleted, whether tests failed
  after the change)
- **flaky_reports** — per-test-node flakiness summary (runs observed, inconsistency count,
  detection timestamp)
- **risk_predictions** — per-file ML risk scores for a commit, plus actual outcome once known
  (for offline evaluation of the model)

## License

MIT