Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@test-intelligence-mcp Check the test coverage of this repo"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
An MCP (Model Context Protocol) server that gives AI coding agents — Claude Code, Claude Desktop, or any other MCP client — the ability to analyse a Python repository's test health: coverage, flaky tests, and ML-based pull-request risk prediction.
Status: under active development. This README grows with each milestone; see Build status below for what's real today vs. what's still coming.
Point it at a Python repository and, from a natural-language conversation with an MCP-aware agent, you can:
Run the repo's test suite with coverage and get real, per-file numbers back (analyze_coverage)
Run the suite multiple times and detect genuinely flaky tests, as opposed to
order-dependent or environment-dependent failures (detect_flaky_tests)
Persist test run results to Postgres to build a history over time (record_test_run)
Query that history back out (get_test_history)
Train a gradient-boosted classifier on accumulated run history to predict which files in a
pull request are likely to break tests (train_risk_model)
Diff a branch against a base ref and get a ranked risk score per changed file
(predict_pr_risk)
Everything is backed by real subprocess test execution and real parsing of coverage.json —
nothing here scrapes terminal output or fakes numbers.
Related MCP server: Chisel
MCP is a protocol (open-sourced by Anthropic, now broadly adopted) that lets an AI agent
discover and call tools exposed by a separate server process, over a standard JSON-RPC
transport (stdio locally, or HTTP/SSE remotely). Instead of hand-rolling a custom API and
teaching an agent's system prompt about it, you expose typed Python functions as "tools"; the
client discovers their names, argument schemas, and docstrings automatically and calls them
mid-conversation. This project uses FastMCP, the
ergonomic Python SDK built on top of the official MCP spec — @mcp.tool() on a normal typed
function is enough to expose it.
Concern | Choice |
MCP server | |
Test execution | pytest, pytest-cov, coverage.py (parses |
Database | PostgreSQL, async SQLAlchemy 2.0 ( |
Migrations | Alembic (versioned, no |
ML | scikit-learn |
Git operations | GitPython / subprocess |
CI | GitHub Actions |
Local Postgres | Docker + docker-compose |
Package management |
test-intelligence-mcp/
src/test_intelligence/
server.py # FastMCP server + tool registration
config.py # typed settings, loaded from .env
safety.py # repo-path allowlist gate (see Safety below)
paths.py # cross-platform file-path normalization
runners/ # pytest/coverage execution, JUnit + coverage.json parsing
flaky/ # multi-run comparison logic, order/seed control
ml/ # features.py, synthetic.py, training.py, prediction.py, model_store.py
db/ # SQLAlchemy models, session, query helpers
git/ # commit/branch metadata (repo_info.py), diff stats (diff.py)
tests/ # tests for THIS project's own code
fixtures/ # tiny throwaway repos the runner tests execute for real
scripts/
ci_report.py # flaky-check + coverage summary, invoked by CI (see below)
alembic/ # migration scripts
.github/workflows/
ci.yml # runs on every PR — see Continuous Integration below
docker-compose.yml # local Postgres
pyproject.toml
.env.examplePython 3.11+
uv — a fast, modern replacement
for pip + venv + virtualenv, used here for dependency management and running commands. On
Windows: winget install -e --id astral-sh.uv.
Docker Desktop — used to run Postgres
locally via docker-compose, so you don't need Postgres installed on your machine. On
Windows: winget install -e --id Docker.DockerDesktop.
uv syncuv sync reads pyproject.toml, resolves a locked dependency set (writing/using
uv.lock), and creates a .venv/ — the uv equivalent of pip install -r requirements.txt
inside a fresh virtualenv, but faster and reproducible across machines.
docker-compose up -dThis starts a Postgres 16 container defined in docker-compose.yml, exposed on
localhost:5433 with the credentials baked into that file. (Port 5433, not the
Postgres-default 5432, to sidestep a collision if you already have Postgres installed
natively — see docker-compose.yml for details.) -d runs it detached (in the
background). Check it's healthy with:
docker-compose psYou should see test-intelligence-postgres with status healthy.
cp .env.example .envThe defaults in .env.example already match docker-compose.yml's credentials, so for local
dev you typically don't need to change anything except TI_ALLOWED_REPO_ROOTS (see
Safety below).
uv run alembic upgrade headAlembic replays every migration script under alembic/versions/ in order, bringing the
database schema up to the latest version. Unlike SQLAlchemy's Base.metadata.create_all()
(which can only stamp out tables matching whatever the current model code says, with no
memory of past states), Alembic tracks schema history as an ordered chain of scripts — so
changes are reviewable in git, reversible (alembic downgrade), and applied identically in
dev, CI, and prod.
claude mcp add test-intelligence -- uv run --directory "C:\path\to\test-intelligence-mcp" test-intelligence-mcpUsing --directory (rather than relying on whatever directory you happened to run
claude mcp add from) makes the registration work regardless of where Claude Code's
process actually launches the server from later — important because the server reads
.env relative to its working directory at startup.
This registers the server as a stdio-transport MCP server scoped to your local Claude Code
config. Verify it's connected with claude mcp list, then start a new Claude Code
session (a session already running won't pick up a server registered after it started)
and ask it to list available tools.
Add to claude_desktop_config.json (Windows:
%APPDATA%\Claude\claude_desktop_config.json):
{
"mcpServers": {
"test-intelligence": {
"command": "uv",
"args": ["--directory", "C:\\path\\to\\test-intelligence-mcp", "run", "test-intelligence-mcp"]
}
}
}Restart Claude Desktop; the six tools should appear under the 🔨 tools icon.
Because these tools execute a target repository's real test suite — i.e. arbitrary Python code — as a subprocess, two guardrails are enforced unconditionally:
Path allowlist: every repo_path argument is resolved to an absolute path and checked
against TI_ALLOWED_REPO_ROOTS (a comma-separated list of permitted base directories) in
.env. Paths outside the allowlist are rejected before any subprocess runs.
Subprocess timeouts: every subprocess call (a pytest run, a git command) has a hard
timeout (TI_SUBPROCESS_TIMEOUT_SECONDS in .env, default 300s) so a hung or
infinite-looping suite can't block the server indefinitely.
analyze_coverageAsk an MCP-aware agent something like: "Run analyze_coverage on
C:\path\to\some-repo". The tool runs that repo's test suite with coverage (using its
own .venv/venv if it has one, otherwise falling back to this server's interpreter)
and returns:
{
"status": "ok",
"tests_passed": true,
"overall_coverage_percent": 87.5,
"total_statements": 120,
"total_covered_lines": 105,
"total_uncovered_lines": 15,
"files": [
{
"file": "pkg/calculator.py",
"coverage_percent": 80.0,
"num_statements": 10,
"covered_lines": 8,
"uncovered_line_count": 2,
"uncovered_lines": [12, 13]
}
]
}files is sorted worst-covered first, so an agent can immediately point at the files
most in need of tests. Requires the target repo to have pytest and pytest-cov
installed in whatever Python environment gets resolved.
record_test_run + get_test_history"Record a test run for C:\path\to\some-repo, then show me the history for
pkg/calculator.py" — the first call executes the suite once via pytest's
--junitxml output (so pytest alone is enough in the target repo, no plugin
needed), persists a Repository/TestRun/TestResult row set to Postgres, and tags
the run with the target repo's current commit SHA and branch (via GitPython) when
it's a real git repo:
{
"status": "ok",
"run_id": 3,
"repo_id": 1,
"commit_sha": "a1b2c3d...",
"branch": "main",
"duration_seconds": 0.52,
"total_tests": 3,
"passed_count": 1,
"failed_count": 1,
"skipped_count": 1
}get_test_history only ever reads rows already written by record_test_run — it
never triggers a run itself, and it queries across every repo this server has ever
recorded (there's no repo_path argument), optionally filtered to one file_path:
{
"status": "ok",
"count": 2,
"history": [
{
"repo_name": "C:\\path\\to\\some-repo",
"run_id": 3,
"commit_sha": "a1b2c3d...",
"branch": "main",
"started_at": "2026-08-16T00:20:11+00:00",
"node_id": "tests/test_calculator.py::test_divide",
"file_path": "tests/test_calculator.py",
"outcome": "passed",
"duration_seconds": 0.001,
"error_message": null
}
]
}detect_flaky_tests"Run detect_flaky_tests on C:\path\to\some-repo with 5 runs" — runs the suite runs
times with known test-order-randomizing plugins (pytest-randomly, pytest-random-order)
explicitly disabled, so test order is identical every run. That isolates genuine
non-determinism (timing, shared state, unseeded randomness in the code under test)
as the only possible explanation for a test disagreeing with itself across runs —
an order-shuffling plugin would otherwise make order-dependent failures
indistinguishable from real flakiness. Progress streams live via MCP progress
notifications (visible to clients that support them) since 5+ sequential runs on a
large suite can take a while:
{
"status": "ok",
"repo_id": 2,
"runs_requested": 5,
"runs_completed": 5,
"total_tests_observed": 2,
"flaky_test_count": 1,
"flaky_tests": [
{
"node_id": "tests/test_flaky.py::test_alternates",
"runs_observed": 5,
"inconsistency_count": 2,
"flakiness_rate": 0.4,
"outcomes": ["passed", "failed", "passed", "failed", "passed"],
"majority_outcome": "passed"
}
],
"run_failures": []
}Detected flaky tests are also persisted to the flaky_reports table.
train_risk_model"Train the risk model" — trains a GradientBoostingClassifier to predict "will a
test fail after this file changes?" from 10 features per file (churn, historical
failure count, current coverage, test count touching the file, days since last
modified, distinct authors, file size, cyclomatic complexity — see
Cold-start strategy for where the training
data comes from), evaluates on a held-out split, and reports honestly:
{
"status": "ok",
"model_path": "models/risk_model.joblib",
"real_sample_count": 0,
"synthetic_sample_count": 500,
"total_sample_count": 500,
"test_set_size": 125,
"metrics": {
"accuracy": 0.6,
"precision": 0.5962,
"recall": 0.5167,
"f1": 0.5536
},
"caveat": "Only 0 real training example(s) recorded so far (via record_test_run) — this training run is dominated by synthetic, artificially-generated bootstrap data. These metrics describe how well the model fits that synthetic relationship, NOT real predictive power on an actual repository. Keep calling record_test_run on real repos, then retrain, before trusting these numbers for anything beyond confirming the training pipeline itself works."
}The caveat field only disappears once real_sample_count clears a real threshold
(30, see ml/training.py) — this tool never presents synthetic-dominated metrics as
if they were validated against reality.
(Remaining tools filled in per-milestone — see Build status.)
train_risk_model needs labelled examples — "given these features about a file
change, did a test tied to that file fail afterward?" On a freshly set up server,
zero runs have been recorded, so there's no history to learn from. Three options
were weighed before writing any ML code:
Replay a real open-source repo's git/CI history. Clone a real project, walk its commits, check each one out, install its dependencies as they existed at that point in history, run its suite, extract real features and real labels. The most realistic data by far — but expensive and fragile to build reliably: dependency installation breaks across years of history (deprecated packages, Python version drift), full-history checkouts are slow, and it bolts a hard external dependency (a specific repo, at a specific point in time) onto this project's own CI, which would need to reproduce it on every run.
Replay this project's own commits. Same idea, smaller scope — doesn't avoid the core cost problem, and this project's own history is far too short and narrow to represent the breadth of file-change patterns a general-purpose risk model should generalize across.
Generate synthetic data (chosen). Draw feature vectors from plausible distributions and derive labels from a deliberately-designed, domain-informed generative rule — more churn + more past failures + lower coverage + higher complexity → higher failure probability, plus noise — rather than a coin flip. Fast, fully reproducible, no external repo required, and enough to exercise the entire pipeline (feature extraction → training → evaluation) honestly today.
The honest tradeoff: a model trained purely on synthetic data has learned the
shape of a plausible risk relationship, not the real one. Its metrics on
held-out synthetic data look reasonable (~0.6 accuracy, ROC-AUC ~0.67 — see
tests/ml/test_synthetic.py), which only proves the pipeline works, not that it
predicts anything about a real repository. train_risk_model blends in real
examples the moment any exist (via record_test_run → file_changes rows — see
below) and always reports the real/synthetic split plus an explicit caveat when
real data is too thin to trust, rather than ever presenting synthetic-derived
numbers as validated.
Where real examples come from: record_test_run computes a real git diff
(HEAD~1..HEAD) after every run and writes one file_changes row per changed
file, labelled tests_failed_after = "did any test fail in this run" — applied
to every file changed in it, not per-file attribution. That's a deliberate choice:
attributing a failure to the specific file that caused it would need
coverage-based tracing (which test executed which source lines), which this
project doesn't do. The coarser signal is honestly correlational ("this file was
part of a commit that broke something"), not causal — see the comment in
runners/record_run.py for the full reasoning, including why a
file-path-string-match heuristic would have looked more precise while actually
being narrower and more misleading.
Historical feature extraction (used for real training examples) reads git history
"as of" each recorded run's own timestamp and commit — git log --before, git show <sha>:<path> — never the file's current state, so a model can't
accidentally train on information that didn't exist yet at prediction time.
coverage_percent is the one feature that can't be reconstructed historically
without re-running the full suite at that exact commit (too expensive to do per
training example), so it's stored as an explicit "unknown" sentinel for real
historical rows, and only computed fresh for live predictions (predict_pr_risk,
below).
predict_pr_risk"Predict PR risk for C:\path\to\some-repo against main" — diffs base_ref..HEAD
(a real git diff --numstat), extracts live features for each changed file
(current working-tree state, plus a fresh analyze_coverage run for real current
coverage — not the "unknown" sentinel historical training rows get), scores each
with the trained model, and ranks highest-risk first:
{
"status": "ok",
"repo_id": 3,
"base_ref": "0bcbeba860df0457c55ad3c1d3826ed5fd941506",
"commit_sha": "8306f4598275f92907de89e1161f982772f3aac7",
"model_trained_at": "2026-08-17T19:03:42.707577+00:00",
"model_real_sample_count": 0,
"predictions": [
{ "file": "tests/test_calculator.py", "predicted_risk_probability": 0.0743, "lines_added": 9, "lines_deleted": 1 },
{ "file": "pkg/calculator.py", "predicted_risk_probability": 0.0457, "lines_added": 7, "lines_deleted": 0 }
]
}model_real_sample_count is carried through from the model's training metadata —
so a caller can see at a glance whether these predictions come from a
synthetic-dominated model (see Cold-start strategy)
without a separate lookup. Requires train_risk_model to have been run at least
once (no_trained_model error otherwise) — this tool never trains a model
implicitly as a side effect. Every prediction is persisted to risk_predictions
with actual_outcome left NULL, so a real repo's predictions can eventually be
checked against what actually happened — that evaluation isn't built yet, but the
data is captured from day one so it can be added without a schema change.
GitHub Actions is CI built directly into GitHub: a workflow — one YAML file,
.github/workflows/ci.yml — describes jobs that run
automatically in response to repo events (here: opening/updating a pull request, or
pushing to main). Each job runs on a fresh, throwaway virtual machine (a
"runner") — nothing persists between runs except what's explicitly cached or
uploaded — and is just a sequence of steps, each either a shell command or a
reusable action (a packaged step someone else published, referenced like
actions/checkout@v4).
This project's workflow is the project testing itself, end to end:
Check out the repo and install uv + dependencies — same tools a
contributor installs locally.
Start Postgres as a service container — a second container GitHub Actions
runs alongside the job, reachable at localhost:5433 from every step, exactly
like docker-compose up -d locally but managed by GitHub instead of Docker
Desktop. The job's steps don't start until its healthcheck passes — no
hand-written "wait for Postgres" polling loop needed.
Apply Alembic migrations, then run this project's own test suite with
--cov-fail-under=$COVERAGE_THRESHOLD — pytest-cov's built-in gate; the build
fails outright if coverage drops below it (currently 80%, with headroom under
the real ~93%).
Run detect_flaky_tests against this project's own tests — via
scripts/ci_report.py, which calls the tool through the
real MCP layer (fastmcp.Client talking to the actual server object), not a
shortcut. Informational only — it never fails the build, only coverage does.
Upload both reports as workflow artifacts (actions/upload-artifact) —
downloadable from the workflow run's page for 90 days by default.
Write a job summary ($GITHUB_STEP_SUMMARY, rendered as Markdown directly
on the run's page) and post it as a PR comment (actions/github-script,
using the run's built-in GITHUB_TOKEN — no extra secrets needed). The step
summary is the fallback that always works, including for PRs from forks, which
get a read-only token that can't post comments (a GitHub security restriction,
not a bug in this workflow) — the comment step is wrapped in
continue-on-error: true so that limitation degrades gracefully instead of
failing the whole job.
To actually see this run, the project needs to live in a real GitHub repository with commits pushed to it — nothing in this local build process has created one yet. Once that exists: open a PR, and the Actions tab (and the PR itself, once the comment lands) shows it running live.
This project is built milestone-by-milestone, each one verified working before moving to the next.
Milestone 1 — project skeleton, docker-compose Postgres, .env.example, this README
Milestone 2 — database layer (SQLAlchemy models + Alembic)
Milestone 3 — FastMCP server skeleton (6 tools registered, placeholder bodies)
Milestone 4 — analyze_coverage
Milestone 5 — record_test_run + get_test_history
Milestone 6 — detect_flaky_tests
Milestone 7 — ML cold-start strategy, feature extraction, train_risk_model
Milestone 8 — predict_pr_risk
Milestone 9 — GitHub Actions CI (built + locally verified; live PR run pending a real GitHub repo)
Milestone 10 — final polish
uv sync --extra dev # installs pytest-asyncio + ruff on top of the base deps
uv run pytest -v
uv run pytest --cov --cov-report=term-missing # with coverage
uv run ruff check . # lintThe DB-layer tests use a real, disposable Postgres database (test_intelligence_test,
created and torn down automatically) and run the actual Alembic migrations against it,
rather than mocking the database or using create_all() — the same approach CI uses
via its Postgres service container (Milestone 9). See tests/conftest.py for details.
Six tables, managed by Alembic migrations:
repositories — a tracked repo (name + local path or remote URL)
test_runs — one row per pytest invocation (repo, commit SHA, branch, timestamp,
duration, pass/fail/skip counts)
test_results — one row per test node ID within a run (outcome, duration, error message)
file_changes — per-file diff stats for a run (lines added/deleted, whether tests failed after the change)
flaky_reports — per-test-node flakiness summary (runs observed, inconsistency count, detection timestamp)
risk_predictions — per-file ML risk scores for a commit, plus actual outcome once known (for offline evaluation of the model)
MIT
Run the test suite with coverage and report real, per-file coverage numbers.
Executes pytest with pytest-cov against the target repository, then parses the machine-readable coverage.json report it produces (never the human-readable terminal summary) to compute overall coverage percentage, per-file coverage, and uncovered line counts.
| Name | Required | Description | Default |
|---|---|---|---|
| repo_path | Yes | Absolute path to the Python repository to analyse. Must resolve to a location under one of this server's allowed repo roots. | |
| test_path | No | Optional path, relative to repo_path, restricting which tests to run (e.g. "tests/unit"). Defaults to the whole test suite. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses that the tool executes pytest with pytest-cov, parses coverage.json (never the human-readable summary), and computes overall coverage, per-file coverage, and uncovered line counts. This is comprehensive for a coverage analysis tool, though it could mention any destructive effects or file modifications.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise at two sentences: the first line states the core purpose, and the second line details the execution and parsing behavior. Every sentence adds value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists (so return values are documented separately), the description adequately covers the tool's behavior, parameters, and execution context. It could mention prerequisites like pytest-cov installation, but for a tool with good schema coverage and output schema, this is mostly sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage, so the baseline is 3. The description adds value by explaining that test_path restricts which tests to run and defaults to the whole suite, and repo_path must be under allowed roots. However, it does not elaborate on the parameters further beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that this tool runs the test suite with coverage and reports real, per-file coverage numbers. It specifies the resource (test suite) and verb (analyze coverage), and the detail about parsing machine-readable coverage.json distinguishes it from any generic coverage tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use this tool (for running pytest with pytest-cov and getting machine-readable per-file coverage), but it does not explicitly say when not to use it or mention alternatives like detect_flaky_tests or record_test_run. It implies usage for coverage analysis among siblings focused on flaky detection, risk prediction, and history.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run the test suite multiple times and flag tests with inconsistent outcomes.
Runs the full suite runs times, comparing each test's outcome (by its stable
pytest node ID) across runs. A test that passes in some runs and fails in
others is reported as flaky, along with its flakiness rate. Test order and
random seeds are controlled so genuine flakiness isn't confused with
order-dependent or environment-dependent failures.
| Name | Required | Description | Default |
|---|---|---|---|
| runs | No | How many times to run the suite. Higher values give more confidence but take proportionally longer. Defaults to 5. | |
| repo_path | Yes | Absolute path to the Python repository to analyse. Must resolve to a location under one of this server's allowed repo roots. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations were provided, so the description carries the full burden. It discloses that the tool runs the full suite multiple times (potentially expensive), uses stable pytest node IDs, controls test order and seeds, and reports flakiness rates. However, it does not mention if the tool modifies any files or has any side effects beyond reporting.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and structured effectively. The first sentence immediately states the core purpose, followed by a short paragraph explaining the mechanism and guarantees. Every sentence adds essential information without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that the tool has only 2 parameters with 100% schema coverage, an output schema exists, and no annotations are provided, the description sufficiently covers what the tool does, how it works, and what guarantees it provides (controlled order/seeds). There are no obvious gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds value by explaining that runs controls the number of executions and that higher values give more confidence but take proportionally longer. It also clarifies that repo_path must be a Python repository and resolves under allowed roots. The first sentence of the description contextualizes both parameters together.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose using a specific verb-resource pair: 'Run the test suite multiple times and flag tests with inconsistent outcomes.' It immediately distinguishes itself from siblings by focusing on flakiness detection rather than coverage analysis, risk prediction, or history retrieval.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implicitly suggests when to use this tool (to identify flaky tests) vs alternatives (e.g., analyze_coverage for coverage, get_test_history for historical results). It explains that test order and seeds are controlled to avoid false positives, but does not explicitly state when NOT to use it or provide direct comparisons to siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Query previously recorded test run history from the database.
Reads rows written by record_test_run — this tool never triggers a new test run itself, it only reports on what's already been recorded.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of history entries to return, most recent first. Defaults to 50. | |
| file_path | No | Optional file path to filter history down to test runs/results touching that specific file. Omit to get history across the whole repo. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description correctly identifies the tool as read-only ('never triggers a new test run', 'reports on what's already been recorded'). This adequately covers the main behavioral trait. However, it could add more details like handling of empty results or performance implications for large histories.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the main purpose, and no wasted words. The description efficiently conveys the tool's role and its safety characteristic.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read-only query tool with an output schema, the description covers purpose, read-only nature, and relationship to record_test_run. It doesn't mention ordering (most recent first) but that is in the parameter description. Minor gap in not stating that history is returned sorted, but overall complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with both parameters fully described. The description adds no additional parameter semantics beyond the schema, so baseline 3 is appropriate. The context about reading history is helpful but does not directly enhance parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it queries previously recorded test run history, with the verb 'query' and resource 'test run history'. It distinguishes from siblings by noting it reads rows written by record_test_run and never triggers a new test run, differentiating it from record_test_run and other tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states this tool is for reading existing history and never triggers a new test run, guiding the agent to use it when querying past results. It indirectly points to record_test_run as the alternative for writing, providing clear usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Diff the current branch against a base ref and rank changed files by predicted risk.
Computes the file-level diff between HEAD and base_ref, extracts the same features the risk model was trained on for each changed file (churn, historical failure count, current coverage, etc.), and runs them through the trained GradientBoostingClassifier to produce a ranked risk score per file.
| Name | Required | Description | Default |
|---|---|---|---|
| base_ref | No | The git ref to diff the current branch against, e.g. "main" or a commit SHA. Defaults to "main". | main |
| repo_path | Yes | Absolute path to the Python repository to analyse. Must resolve to a location under one of this server's allowed repo roots. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral burden. It explains the internal steps (diffing, feature extraction, model inference) and names the model type (GradientBoostingClassifier), which aids transparency. However, it does not disclose side effects (e.g., does it write anything) or failure modes for missing refs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences: the first is a concise summary, the second provides helpful internal detail. Each sentence adds value, though the second could be slightly tighter.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity (2 params, 100% schema coverage, and an output schema exists), the description is largely sufficient. It explains how the tool works internally, which compensates for the lack of annotations, leaving no major gaps for typical use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents both parameters. The description calls out 'base_ref' implicitly by stating it diffs HEAD against base_ref, but does not add new semantics beyond the schema's default and description for base_ref.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool diffs the current branch against a base ref and ranks changed files by predicted risk. It uses specific verbs ('diff', 'rank') and resources ('changed files', 'predicted risk'), distinguishing it from siblings like 'analyze_coverage' or 'train_risk_model'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is used for assessing risk of changed files in a branch, which differs from siblings like 'detect_flaky_tests' or 'analyze_coverage'. It does not explicitly mention when not to use it or provide alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Execute the test suite once and persist the results to the database.
This is the explicit, distinct step that builds up the training data other tools (get_test_history, train_risk_model) rely on — it never runs implicitly as a side effect of another tool, so it's always clear when history is being written versus just read or analysed.
| Name | Required | Description | Default |
|---|---|---|---|
| repo_path | Yes | Absolute path to the Python repository to run. Must resolve to a location under one of this server's allowed repo roots. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, description carries full burden. It discloses that the tool executes a test suite (potentially side-effecting) and persists results, and it emphasizes the explicit, distinct nature preventing implicit side effects. It does not mention failure behavior or data overwrite semantics, but it provides security context (repo_path must resolve under allowed roots).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, first is direct action, second adds valuable context on when to use. No redundant filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool executes arbitrary Python tests and persists data, description covers the core role and relation to siblings. Output schema exists, so return values are documented elsewhere. Missing some edge-case behavior but adequate for an AI agent to select and invoke.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers the only parameter fully with a description including absolute path and allowed root constraint. The tool description does not add further meaning, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description starts with a specific action: 'Execute the test suite once and persist the results to the database.' It clearly distinguishes from siblings by explaining that this is the explicit write step while get_test_history and train_risk_model consume the data, and it never runs implicitly, so it's unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explains that this tool is the explicit step for writing history versus reading or analyzing, referencing sibling tools get_test_history and train_risk_model. It could be more explicit about when NOT to use it (e.g., for read-only queries, use get_test_history), but the contrast between 'written versus just read or analysed' provides clear context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Retrain the PR-risk classifier on everything currently in the database.
Trains a GradientBoostingClassifier using file_changes/test_results history as labelled examples (did a test tied to this file fail within N runs after the file changed?), then reports accuracy, precision, recall, and F1 on a held-out split. When the training set is small, the returned metrics come with an explicit caveat that they aren't yet statistically meaningful — this tool never reports metrics as trustworthy without that context.
Returns: Training set size and accuracy/precision/recall/F1, with an honest caveat attached when the training set is too small for the metrics to mean much.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully explains the tool's behavior: it trains a GradientBoostingClassifier, uses training history, reports metrics, and includes a caveat for small datasets. The disclosure about statistical meaningfulness is valuable. However, it doesn't mention how long training might take, whether it requires a running database, or if it's a destructive operation (overwrites the current model?), which would justify a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three well-structured paragraphs: first a one-liner summary, then technical detail, then returns section. Every sentence adds value without redundancy. It is concise (every sentence earns its place) and front-loaded with the core purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The output schema is present, so return values are documented elsewhere. The description covers the algorithm, training data source, metric reporting with caveats, and honesty about statistical significance. This is complete for a training-tool description given the schema availability and zero parameters.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, and schema description coverage is 100% (schema is empty). The description doesn't need to explain parameters, but it adds value by detailing the training process and metrics returned. Baseline is 4 due to no parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Retrain the PR-risk classifier on everything currently in the database') and specifies the algorithm (GradientBoostingClassifier) and output metrics. It distinguishes the tool from siblings like 'predict_pr_risk' (which presumably uses the model, not trains it) and 'analyze_coverage' (which analyzes coverage, not risk).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is for retraining the model using historical data, with no explicit when-to-use or when-not-to-use guidance. However, given that all sibling tools have distinct purposes (e.g., 'predict_pr_risk' for predictions, 'analyze_coverage' for coverage), the context is clear enough. No alternatives or exclusions are stated, so score 4.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v0.1.0analyze_coveragedetect_flaky_testsget_test_historypredict_pr_riskrecord_test_runtrain_risk_modelEach tool targets a distinct function: coverage analysis, flaky detection, risk prediction, test execution recording, history querying, and model training. There is no functional overlap, and the descriptions clearly differentiate their purposes.
Tool names consistently use a verb_noun pattern (e.g., analyze_coverage, detect_flaky_tests, train_risk_model). The pattern is clear and predictable, though 'record_test_run' and 'get_test_history' slightly shift the pattern (record vs. get, test_run vs. test_history), but it's still minor and readable.
With 6 tools, the set is well-scoped for a CI/testing intelligence server. Each tool covers a necessary stage (record, query, analyze, detect flakiness, predict risk, train model), and none feel redundant or extraneous.
The tool set covers the core lifecycle: recording test runs (record_test_run), querying history (get_test_history), analyzing coverage (analyze_coverage), detecting flakiness (detect_flaky_tests), predicting risk (predict_pr_risk), and training the risk model (train_risk_model). A minor gap is the lack of a tool to delete or manage stored history, but the essential pipeline is complete.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
An MCP server that gives your AI access to the source code and docs of all public github repos
MCP server for building and testing AI agents with multi-model experimentation and insights.
Nifty's MCP server — exposes tasks, projects, messages, and files as tools for AI agents.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/shreyasKaturi2004/test-intelligence-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
