一个 MCP（模型上下文协议）服务器，为 AI 编码代理（Claude Code、Claude Desktop 或任何其他 MCP 客户端）提供分析 Python 仓库测试健康状况的能力：覆盖率、不稳定测试以及基于机器学习的拉取请求风险预测。
状态： 正在积极开发中。本 README 随每个里程碑更新；请参阅下面的构建状态了解当前实际功能与即将推出的功能。
将其指向一个 Python 仓库，通过与支持 MCP 的代理进行自然语言对话，您可以：
运行仓库的测试套件并获取覆盖率，返回每个文件的真实数字（analyze_coverage）
多次运行测试套件，检测真正不稳定的测试，而非依赖于顺序或环境导致的失败（detect_flaky_tests）
将测试运行结果持久化到 Postgres，以建立历史记录（record_test_run）
查询历史记录（get_test_history）
在累积的运行历史基础上训练梯度提升分类器，预测拉取请求中哪些文件可能破坏测试（train_risk_model）
比较分支与基础引用，获取每个更改文件的排名风险评分（predict_pr_risk）
所有功能都基于真实的子进程测试执行和真实的 coverage.json 解析——这里没有抓取终端输出或伪造数字。
Related MCP server: Chisel
MCP 是一个协议（由 Anthropic 开源，现已广泛采用），允许 AI 代理发现并调用由独立服务器进程通过标准 JSON-RPC 传输（本地使用 stdio，远程使用 HTTP/SSE）暴露的工具。无需手动构建自定义 API 并在代理的系统提示中教导它，您可以将类型化的 Python 函数暴露为“工具”；客户端自动发现它们的名称、参数模式和文档字符串，并在对话过程中调用它们。本项目使用 FastMCP，这是基于官方 MCP 规范构建的符合人体工程学的 Python SDK——在普通的类型化函数上使用 @mcp.tool() 就足以暴露它。
关注点 | 选择 |
MCP 服务器 | |
测试执行 | pytest, pytest-cov, coverage.py（解析 |
数据库 | PostgreSQL, 异步 SQLAlchemy 2.0 ( |
迁移 | Alembic（版本化，无 |
机器学习 | scikit-learn |
Git 操作 | GitPython / subprocess |
CI | GitHub Actions |
本地 Postgres | Docker + docker-compose |
包管理 |
test-intelligence-mcp/
src/test_intelligence/
server.py # FastMCP server + tool registration
config.py # typed settings, loaded from .env
safety.py # repo-path allowlist gate (see Safety below)
paths.py # cross-platform file-path normalization
runners/ # pytest/coverage execution, JUnit + coverage.json parsing
flaky/ # multi-run comparison logic, order/seed control
ml/ # features.py, synthetic.py, training.py, prediction.py, model_store.py
db/ # SQLAlchemy models, session, query helpers
git/ # commit/branch metadata (repo_info.py), diff stats (diff.py)
tests/ # tests for THIS project's own code
fixtures/ # tiny throwaway repos the runner tests execute for real
scripts/
ci_report.py # flaky-check + coverage summary, invoked by CI (see below)
alembic/ # migration scripts
.github/workflows/
ci.yml # runs on every PR — see Continuous Integration below
docker-compose.yml # local Postgres
pyproject.toml
.env.examplePython 3.11+
uv——一个快速、现代的 pip + venv + virtualenv 替代品，用于依赖管理和运行命令。在 Windows 上：winget install -e --id astral-sh.uv。
Docker Desktop——用于通过 docker-compose 在本地运行 Postgres，因此您无需在机器上安装 Postgres。在 Windows 上：winget install -e --id Docker.DockerDesktop。
uv syncuv sync 读取 pyproject.toml，解析锁定的依赖集（写入/使用 uv.lock），并创建 .venv/——相当于在全新的虚拟环境中执行 pip install -r requirements.txt，但速度更快且跨机器可重现。
docker-compose up -d这将启动 docker-compose.yml 中定义的 Postgres 16 容器，暴露在 localhost:5433，凭据嵌入在该文件中。（端口 5433，而非 Postgres 默认的 5432，以避免与您本地已安装的 Postgres 冲突——详见 docker-compose.yml。）-d 参数使其在后台运行。检查其健康状况：
docker-compose ps您应该看到 test-intelligence-postgres 的状态为 healthy。
cp .env.example .env.env.example 中的默认值已与 docker-compose.yml 的凭据匹配，因此对于本地开发，通常无需更改任何内容，除了 TI_ALLOWED_REPO_ROOTS（请参阅下面的安全性）。
uv run alembic upgrade headAlembic 按顺序重放 alembic/versions/ 下的每个迁移脚本，将数据库模式更新到最新版本。与 SQLAlchemy 的 Base.metadata.create_all()（只能根据当前模型代码创建匹配的表，不记忆过去的状态）不同，Alembic 将模式历史记录为有序的脚本链——因此更改可在 git 中审查、可回滚（alembic downgrade），并在开发、CI 和生产环境中一致应用。
claude mcp add test-intelligence -- uv run --directory "C:\path\to\test-intelligence-mcp" test-intelligence-mcp使用 --directory（而不是依赖您运行 claude mcp add 时的当前目录）可使注册在任何目录下都能工作，因为 Claude Code 的进程稍后可能从其他位置启动服务器——这一点很重要，因为服务器在启动时会相对于其工作目录读取 .env。
这将服务器注册为 stdio 传输的 MCP 服务器，作用域限定在您的本地 Claude Code 配置中。使用 claude mcp list 验证连接，然后启动一个新的 Claude Code 会话（已在运行的会话不会拾取在它之后注册的服务器），并要求它列出可用工具。
添加到 claude_desktop_config.json（Windows：%APPDATA%\Claude\claude_desktop_config.json）：
{
"mcpServers": {
"test-intelligence": {
"command": "uv",
"args": ["--directory", "C:\\path\\to\\test-intelligence-mcp", "run", "test-intelligence-mcp"]
}
}
}重启 Claude Desktop；六个工具应出现在 🔨 工具图标下。
由于这些工具会以子进程方式执行目标仓库的真实测试套件（即任意 Python 代码），因此无条件强制执行两个防护措施：
路径白名单：每个 repo_path 参数都会被解析为绝对路径，并根据 .env 中的 TI_ALLOWED_REPO_ROOTS（允许的基础目录的逗号分隔列表）进行检查。白名单之外的路径会在任何子进程运行之前被拒绝。
子进程超时：每个 subprocess 调用（pytest 运行、git 命令）都有硬性超时（.env 中的 TI_SUBPROCESS_TIMEOUT_SECONDS，默认 300 秒），因此挂起或无限循环的测试套件不会无限期阻塞服务器。
analyze_coverage向支持 MCP 的代理提问，例如：“对 C:\path\to\some-repo 运行 analyze_coverage”。该工具会运行该仓库的测试套件并启用覆盖率（如果仓库有自己的 .venv/venv，则使用它；否则回退到此服务器的解释器），并返回：
{
"status": "ok",
"tests_passed": true,
"overall_coverage_percent": 87.5,
"total_statements": 120,
"total_covered_lines": 105,
"total_uncovered_lines": 15,
"files": [
{
"file": "pkg/calculator.py",
"coverage_percent": 80.0,
"num_statements": 10,
"covered_lines": 8,
"uncovered_line_count": 2,
"uncovered_lines": [12, 13]
}
]
}files 按覆盖率从低到高排序，因此代理可以立即指出最需要测试的文件。要求目标仓库在解析到的 Python 环境中安装了 pytest 和 pytest-cov。
record_test_run + get_test_history“记录 C:\path\to\some-repo 的测试运行，然后显示 pkg/calculator.py 的历史记录”——第一次调用通过 pytest 的 --junitxml 输出执行一次测试套件（因此目标仓库中只需 pytest 即可，无需插件），将 Repository/TestRun/TestResult 行集持久化到 Postgres，并在目标仓库是真实 git 仓库时，使用 GitPython 标记当前提交 SHA 和分支：
{
"status": "ok",
"run_id": 3,
"repo_id": 1,
"commit_sha": "a1b2c3d...",
"branch": "main",
"duration_seconds": 0.52,
"total_tests": 3,
"passed_count": 1,
"failed_count": 1,
"skipped_count": 1
}get_test_history 仅读取 record_test_run 已写入的行——它从不触发运行本身，并且查询此服务器记录的所有仓库（没有 repo_path 参数），可选地按一个 file_path 过滤：
{
"status": "ok",
"count": 2,
"history": [
{
"repo_name": "C:\\path\\to\\some-repo",
"run_id": 3,
"commit_sha": "a1b2c3d...",
"branch": "main",
"started_at": "2026-08-16T00:20:11+00:00",
"node_id": "tests/test_calculator.py::test_divide",
"file_path": "tests/test_calculator.py",
"outcome": "passed",
"duration_seconds": 0.001,
"error_message": null
}
]
}detect_flaky_tests“对 C:\path\to\some-repo 运行 detect_flaky_tests，运行 5 次”——运行测试套件 runs 次，并显式禁用已知的测试顺序随机化插件（pytest-randomly、pytest-random-order），因此每次运行的测试顺序完全相同。这隔离了真正的不确定性（时序、共享状态、被测代码中未种子的随机性）作为测试在不同运行中结果不一致的唯一可能解释——否则顺序随机化插件会使顺序依赖的失败与真正的不稳定难以区分。进度通过 MCP 进度通知实时流式传输（对支持它们的客户端可见），因为在大套件上连续运行 5 次以上可能需要一些时间：
{
"status": "ok",
"repo_id": 2,
"runs_requested": 5,
"runs_completed": 5,
"total_tests_observed": 2,
"flaky_test_count": 1,
"flaky_tests": [
{
"node_id": "tests/test_flaky.py::test_alternates",
"runs_observed": 5,
"inconsistency_count": 2,
"flakiness_rate": 0.4,
"outcomes": ["passed", "failed", "passed", "failed", "passed"],
"majority_outcome": "passed"
}
],
"run_failures": []
}检测到的不稳定测试也会持久化到 flaky_reports 表中。
train_risk_model“训练风险模型”——训练一个 GradientBoostingClassifier，根据每个文件的 10 个特征（变更量、历史失败次数、当前覆盖率、触及该文件的测试数量、上次修改以来的天数、不同作者、文件大小、圈复杂度——请参阅冷启动策略了解训练数据的来源），预测“此文件更改后测试是否会失败”，在保留集上评估，并如实报告：
{
"status": "ok",
"model_path": "models/risk_model.joblib",
"real_sample_count": 0,
"synthetic_sample_count": 500,
"total_sample_count": 500,
"test_set_size": 125,
"metrics": {
"accuracy": 0.6,
"precision": 0.5962,
"recall": 0.5167,
"f1": 0.5536
},
"caveat": "Only 0 real training example(s) recorded so far (via record_test_run) — this training run is dominated by synthetic, artificially-generated bootstrap data. These metrics describe how well the model fits that synthetic relationship, NOT real predictive power on an actual repository. Keep calling record_test_run on real repos, then retrain, before trusting these numbers for anything beyond confirming the training pipeline itself works."
}caveat 字段仅在 real_sample_count 超过真实阈值（30，请参阅 ml/training.py）时消失——此工具永远不会将合成数据主导的指标呈现为经过现实验证的。
（其余工具按里程碑填充——请参阅构建状态。）
train_risk_model 需要带标签的示例——“给定关于文件更改的这些特征，与该文件相关的测试之后是否失败？”在全新设置的服务器上，尚未记录任何运行，因此没有历史可供学习。在编写任何 ML 代码之前，权衡了三种选项：
重放真实开源仓库的 git/CI 历史。 克隆一个真实项目，遍历其提交，检出每个提交，安装该历史时刻存在的依赖，运行其测试套件，提取真实特征和真实标签。这是最真实的数据——但构建可靠代价高昂且脆弱：依赖安装会在多年的历史中中断（已弃用的包、Python 版本漂移），完整历史检出很慢，并且它将一个硬的外部依赖（特定仓库、特定时间点）绑定到本项目自己的 CI 上，而 CI 需要在每次运行时重现它。
重放本项目自己的提交。 相同的想法，范围更小——但并未避免核心成本问题，而且本项目自己的历史太短太窄，无法代表通用风险模型应泛化的文件更改模式的广度。
生成合成数据（已选择）。 从合理的分布中抽取特征向量，并根据精心设计的、领域信息丰富的生成规则（更多变更 + 更多历史失败 + 更低覆盖率 + 更高复杂度 → 更高失败概率，加上噪声）推导标签，而不是抛硬币。快速、完全可重现、无需外部仓库，并且足以在今天诚实地演练整个流水线（特征提取 → 训练 → 评估）。
诚实的权衡： 纯粹在合成数据上训练的模型只学习了合理风险关系的形状，而非真实关系。其在保留合成数据上的指标看起来合理（准确率约 0.6，ROC-AUC 约 0.67——请参阅 tests/ml/test_synthetic.py），这仅证明流水线有效，而非它能预测真实仓库的任何内容。train_risk_model 在真实示例存在时立即混合它们（通过 record_test_run → file_changes 行——见下文），并始终报告真实/合成数据的比例，以及在真实数据太少不可信时给出明确的警告，而不是将合成数据得出的数字呈现为经过验证的。
真实示例的来源： record_test_run 在每次运行后计算真实的 git diff (HEAD~1..HEAD)，并为每个更改的文件写入一行 file_changes，标记为 tests_failed_after = "本次运行中是否有测试失败" — 应用于本次运行中更改的所有文件，而不是按文件归因。这是有意为之的选择：将失败归因于导致失败的特定文件需要基于覆盖率的追踪（哪个测试执行了哪些源代码行），而本项目不进行此操作。这个较粗略的信号实际上只是相关性（"该文件是导致某些问题的提交的一部分"），而非因果关系 — 详见 runners/record_run.py 中的注释，其中包含完整推理，包括为什么文件路径字符串匹配启发式方法看起来更精确，但实际上更狭隘且更具误导性。
历史特征提取（用于真实的训练示例）读取每个记录运行的时间戳和提交"当时"的 git 历史 — git log --before、git show <sha>:<path> — 从不使用文件的当前状态，因此模型不会意外地训练预测时尚未存在的信息。coverage_percent 是无法在没有重新运行完整测试套件的情况下重建的一个特征（对每个训练示例来说成本太高），因此对于真实的历史行，它被存储为显式的"未知"标记，仅对实时预测（predict_pr_risk，如下）进行新鲜计算。
predict_pr_risk"预测 C:\path\to\some-repo 相对于 main 的 PR 风险" — 差异比较 base_ref..HEAD（真实的 git diff --numstat），为每个更改的文件提取实时特征（当前工作树状态，加上一次新鲜的 analyze_coverage 运行以获取真实的当前覆盖率 — 而不是历史训练行使用的"未知"标记），使用训练好的模型进行评分，并按风险从高到低排序：
{
"status": "ok",
"repo_id": 3,
"base_ref": "0bcbeba860df0457c55ad3c1d3826ed5fd941506",
"commit_sha": "8306f4598275f92907de89e1161f982772f3aac7",
"model_trained_at": "2026-08-17T19:03:42.707577+00:00",
"model_real_sample_count": 0,
"predictions": [
{ "file": "tests/test_calculator.py", "predicted_risk_probability": 0.0743, "lines_added": 9, "lines_deleted": 1 },
{ "file": "pkg/calculator.py", "predicted_risk_probability": 0.0457, "lines_added": 7, "lines_deleted": 0 }
]
}model_real_sample_count 从模型的训练元数据中传递 — 因此调用者可以一眼看出这些预测是否来自以合成数据为主的模型（请参阅冷启动策略），而无需单独查找。需要至少运行过一次 train_risk_model（否则会报 no_trained_model 错误）— 此工具永远不隐式地作为副作用训练模型。每次预测都会持久化到 risk_predictions，其中 actual_outcome 保留为 NULL，以便最终可以对照实际结果检查真实仓库的预测 — 尚未构建该评估，但从第一天起就捕获数据，因此可以在不更改模式的情况下添加。
GitHub Actions 是直接构建在 GitHub 中的 CI：一个工作流 — 一个 YAML 文件，.github/workflows/ci.yml — 描述根据仓库事件（此处：打开/更新拉取请求，或推送到 main）自动运行的作业。每个作业在一个全新的、一次性的虚拟机上运行（"runner"）— 除了显式缓存或上传的内容外，运行之间不会持久化任何内容 — 并且只是步骤的序列，每个步骤要么是 shell 命令，要么是可重用的操作（其他人发布的打包步骤，像 actions/checkout@v4 这样引用）。
本项目的工作流是项目自身端到端的测试：
检出仓库并安装 uv + 依赖项 — 与贡献者本地安装的工具相同。
将 Postgres 作为服务容器启动 — 一个与作业并行的容器，GitHub Actions 在每个步骤中均可通过 localhost:5433 访问，就像本地使用 docker-compose up -d 一样，但由 GitHub 而不是 Docker Desktop 管理。作业的步骤在其健康检查通过之前不会启动 — 无需手动编写"等待 Postgres"的轮询循环。
应用 Alembic 迁移，然后使用 --cov-fail-under=$COVERAGE_THRESHOLD 运行本项目自己的测试套件 — pytest-cov 的内置门控；如果覆盖率低于该阈值（目前为 80%，低于实际约 93% 的覆盖率留有空间），构建将直接失败。
对项目自身的测试运行 detect_flaky_tests — 通过 scripts/ci_report.py，该脚本通过真实的 MCP 层（fastmcp.Client 与实际的服务器对象通信）调用工具，不使用快捷方式。仅提供信息 — 它永远不会使构建失败，只有覆盖率会。
将两个报告作为工作流制品上传（actions/upload-artifact）— 默认情况下，可从工作流运行的页面下载 90 天。
写入作业摘要（$GITHUB_STEP_SUMMARY，直接在工作流运行页面上渲染为 Markdown）并将其作为 PR 评论发布（actions/github-script，使用运行的内置 GITHUB_TOKEN — 无需额外机密）。步骤摘要始终有效，包括来自 fork 的 PR，这些 PR 获得的是只读令牌，无法发表评论（这是 GitHub 的安全限制，不是本工作流的错误）— 评论步骤被包裹在 continue-on-error: true 中，因此该限制会优雅降级，而不会导致整个作业失败。
要实际看到此运行，项目需要存在于一个真实的 GitHub 仓库中，并推送提交 — 本地构建过程中尚未创建这样的仓库。一旦存在：打开一个 PR，Actions 选项卡（以及 PR 本身，一旦评论落下）将显示其运行情况。
本项目按里程碑构建，每个里程碑在进入下一个之前验证其工作正常。
里程碑 1 — 项目骨架、docker-compose Postgres、.env.example、本文档
里程碑 2 — 数据库层（SQLAlchemy 模型 + Alembic）
里程碑 3 — FastMCP 服务器骨架（6 个已注册的工具，占位主体）
里程碑 4 — analyze_coverage
里程碑 5 — record_test_run + get_test_history
里程碑 6 — detect_flaky_tests
里程碑 7 — ML 冷启动策略、特征提取、train_risk_model
里程碑 8 — predict_pr_risk
里程碑 9 — GitHub Actions CI（已构建并在本地验证；实时 PR 运行待真实 GitHub 仓库）
里程碑 10 — 最终打磨
uv sync --extra dev # installs pytest-asyncio + ruff on top of the base deps
uv run pytest -v
uv run pytest --cov --cov-report=term-missing # with coverage
uv run ruff check . # lint数据库层测试使用真实的、一次性的 Postgres 数据库（test_intelligence_test，自动创建和销毁），并针对其运行实际的 Alembic 迁移，而不是模拟数据库或使用 create_all() — 与 CI 通过其 Postgres 服务容器（里程碑 9）使用的方法相同。详见 tests/conftest.py。
六个表，由 Alembic 迁移管理：
repositories — 跟踪的仓库（名称 + 本地路径或远程 URL）
test_runs — 每次 pytest 调用一行（仓库、提交 SHA、分支、时间戳、持续时间、通过/失败/跳过计数）
test_results — 每次运行中每个测试节点 ID 一行（结果、持续时间、错误消息）
file_changes — 每次运行的每个文件 diff 统计（添加/删除的行数、更改后测试是否失败）
flaky_reports — 每个测试节点的脆弱性摘要（观察到的运行次数、不一致计数、检测时间戳）
risk_predictions — 每次提交的每个文件 ML 风险评分，以及已知后的实际结果（用于模型离线评估）
MIT
Run the test suite with coverage and report real, per-file coverage numbers.
Executes pytest with pytest-cov against the target repository, then parses the machine-readable coverage.json report it produces (never the human-readable terminal summary) to compute overall coverage percentage, per-file coverage, and uncovered line counts.
| Name | Required | Description | Default |
|---|---|---|---|
| repo_path | Yes | Absolute path to the Python repository to analyse. Must resolve to a location under one of this server's allowed repo roots. | |
| test_path | No | Optional path, relative to repo_path, restricting which tests to run (e.g. "tests/unit"). Defaults to the whole test suite. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses that the tool executes pytest with pytest-cov, parses coverage.json (never the human-readable summary), and computes overall coverage, per-file coverage, and uncovered line counts. This is comprehensive for a coverage analysis tool, though it could mention any destructive effects or file modifications.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise at two sentences: the first line states the core purpose, and the second line details the execution and parsing behavior. Every sentence adds value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists (so return values are documented separately), the description adequately covers the tool's behavior, parameters, and execution context. It could mention prerequisites like pytest-cov installation, but for a tool with good schema coverage and output schema, this is mostly sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage, so the baseline is 3. The description adds value by explaining that test_path restricts which tests to run and defaults to the whole suite, and repo_path must be under allowed roots. However, it does not elaborate on the parameters further beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that this tool runs the test suite with coverage and reports real, per-file coverage numbers. It specifies the resource (test suite) and verb (analyze coverage), and the detail about parsing machine-readable coverage.json distinguishes it from any generic coverage tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use this tool (for running pytest with pytest-cov and getting machine-readable per-file coverage), but it does not explicitly say when not to use it or mention alternatives like detect_flaky_tests or record_test_run. It implies usage for coverage analysis among siblings focused on flaky detection, risk prediction, and history.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run the test suite multiple times and flag tests with inconsistent outcomes.
Runs the full suite runs times, comparing each test's outcome (by its stable
pytest node ID) across runs. A test that passes in some runs and fails in
others is reported as flaky, along with its flakiness rate. Test order and
random seeds are controlled so genuine flakiness isn't confused with
order-dependent or environment-dependent failures.
| Name | Required | Description | Default |
|---|---|---|---|
| runs | No | How many times to run the suite. Higher values give more confidence but take proportionally longer. Defaults to 5. | |
| repo_path | Yes | Absolute path to the Python repository to analyse. Must resolve to a location under one of this server's allowed repo roots. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations were provided, so the description carries the full burden. It discloses that the tool runs the full suite multiple times (potentially expensive), uses stable pytest node IDs, controls test order and seeds, and reports flakiness rates. However, it does not mention if the tool modifies any files or has any side effects beyond reporting.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and structured effectively. The first sentence immediately states the core purpose, followed by a short paragraph explaining the mechanism and guarantees. Every sentence adds essential information without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that the tool has only 2 parameters with 100% schema coverage, an output schema exists, and no annotations are provided, the description sufficiently covers what the tool does, how it works, and what guarantees it provides (controlled order/seeds). There are no obvious gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds value by explaining that runs controls the number of executions and that higher values give more confidence but take proportionally longer. It also clarifies that repo_path must be a Python repository and resolves under allowed roots. The first sentence of the description contextualizes both parameters together.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose using a specific verb-resource pair: 'Run the test suite multiple times and flag tests with inconsistent outcomes.' It immediately distinguishes itself from siblings by focusing on flakiness detection rather than coverage analysis, risk prediction, or history retrieval.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implicitly suggests when to use this tool (to identify flaky tests) vs alternatives (e.g., analyze_coverage for coverage, get_test_history for historical results). It explains that test order and seeds are controlled to avoid false positives, but does not explicitly state when NOT to use it or provide direct comparisons to siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Query previously recorded test run history from the database.
Reads rows written by record_test_run — this tool never triggers a new test run itself, it only reports on what's already been recorded.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of history entries to return, most recent first. Defaults to 50. | |
| file_path | No | Optional file path to filter history down to test runs/results touching that specific file. Omit to get history across the whole repo. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description correctly identifies the tool as read-only ('never triggers a new test run', 'reports on what's already been recorded'). This adequately covers the main behavioral trait. However, it could add more details like handling of empty results or performance implications for large histories.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the main purpose, and no wasted words. The description efficiently conveys the tool's role and its safety characteristic.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read-only query tool with an output schema, the description covers purpose, read-only nature, and relationship to record_test_run. It doesn't mention ordering (most recent first) but that is in the parameter description. Minor gap in not stating that history is returned sorted, but overall complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with both parameters fully described. The description adds no additional parameter semantics beyond the schema, so baseline 3 is appropriate. The context about reading history is helpful but does not directly enhance parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it queries previously recorded test run history, with the verb 'query' and resource 'test run history'. It distinguishes from siblings by noting it reads rows written by record_test_run and never triggers a new test run, differentiating it from record_test_run and other tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states this tool is for reading existing history and never triggers a new test run, guiding the agent to use it when querying past results. It indirectly points to record_test_run as the alternative for writing, providing clear usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Diff the current branch against a base ref and rank changed files by predicted risk.
Computes the file-level diff between HEAD and base_ref, extracts the same features the risk model was trained on for each changed file (churn, historical failure count, current coverage, etc.), and runs them through the trained GradientBoostingClassifier to produce a ranked risk score per file.
| Name | Required | Description | Default |
|---|---|---|---|
| base_ref | No | The git ref to diff the current branch against, e.g. "main" or a commit SHA. Defaults to "main". | main |
| repo_path | Yes | Absolute path to the Python repository to analyse. Must resolve to a location under one of this server's allowed repo roots. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral burden. It explains the internal steps (diffing, feature extraction, model inference) and names the model type (GradientBoostingClassifier), which aids transparency. However, it does not disclose side effects (e.g., does it write anything) or failure modes for missing refs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences: the first is a concise summary, the second provides helpful internal detail. Each sentence adds value, though the second could be slightly tighter.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity (2 params, 100% schema coverage, and an output schema exists), the description is largely sufficient. It explains how the tool works internally, which compensates for the lack of annotations, leaving no major gaps for typical use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents both parameters. The description calls out 'base_ref' implicitly by stating it diffs HEAD against base_ref, but does not add new semantics beyond the schema's default and description for base_ref.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool diffs the current branch against a base ref and ranks changed files by predicted risk. It uses specific verbs ('diff', 'rank') and resources ('changed files', 'predicted risk'), distinguishing it from siblings like 'analyze_coverage' or 'train_risk_model'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is used for assessing risk of changed files in a branch, which differs from siblings like 'detect_flaky_tests' or 'analyze_coverage'. It does not explicitly mention when not to use it or provide alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Execute the test suite once and persist the results to the database.
This is the explicit, distinct step that builds up the training data other tools (get_test_history, train_risk_model) rely on — it never runs implicitly as a side effect of another tool, so it's always clear when history is being written versus just read or analysed.
| Name | Required | Description | Default |
|---|---|---|---|
| repo_path | Yes | Absolute path to the Python repository to run. Must resolve to a location under one of this server's allowed repo roots. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, description carries full burden. It discloses that the tool executes a test suite (potentially side-effecting) and persists results, and it emphasizes the explicit, distinct nature preventing implicit side effects. It does not mention failure behavior or data overwrite semantics, but it provides security context (repo_path must resolve under allowed roots).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, first is direct action, second adds valuable context on when to use. No redundant filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool executes arbitrary Python tests and persists data, description covers the core role and relation to siblings. Output schema exists, so return values are documented elsewhere. Missing some edge-case behavior but adequate for an AI agent to select and invoke.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers the only parameter fully with a description including absolute path and allowed root constraint. The tool description does not add further meaning, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description starts with a specific action: 'Execute the test suite once and persist the results to the database.' It clearly distinguishes from siblings by explaining that this is the explicit write step while get_test_history and train_risk_model consume the data, and it never runs implicitly, so it's unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explains that this tool is the explicit step for writing history versus reading or analyzing, referencing sibling tools get_test_history and train_risk_model. It could be more explicit about when NOT to use it (e.g., for read-only queries, use get_test_history), but the contrast between 'written versus just read or analysed' provides clear context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Retrain the PR-risk classifier on everything currently in the database.
Trains a GradientBoostingClassifier using file_changes/test_results history as labelled examples (did a test tied to this file fail within N runs after the file changed?), then reports accuracy, precision, recall, and F1 on a held-out split. When the training set is small, the returned metrics come with an explicit caveat that they aren't yet statistically meaningful — this tool never reports metrics as trustworthy without that context.
Returns: Training set size and accuracy/precision/recall/F1, with an honest caveat attached when the training set is too small for the metrics to mean much.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully explains the tool's behavior: it trains a GradientBoostingClassifier, uses training history, reports metrics, and includes a caveat for small datasets. The disclosure about statistical meaningfulness is valuable. However, it doesn't mention how long training might take, whether it requires a running database, or if it's a destructive operation (overwrites the current model?), which would justify a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three well-structured paragraphs: first a one-liner summary, then technical detail, then returns section. Every sentence adds value without redundancy. It is concise (every sentence earns its place) and front-loaded with the core purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The output schema is present, so return values are documented elsewhere. The description covers the algorithm, training data source, metric reporting with caveats, and honesty about statistical significance. This is complete for a training-tool description given the schema availability and zero parameters.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, and schema description coverage is 100% (schema is empty). The description doesn't need to explain parameters, but it adds value by detailing the training process and metrics returned. Baseline is 4 due to no parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Retrain the PR-risk classifier on everything currently in the database') and specifies the algorithm (GradientBoostingClassifier) and output metrics. It distinguishes the tool from siblings like 'predict_pr_risk' (which presumably uses the model, not trains it) and 'analyze_coverage' (which analyzes coverage, not risk).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is for retraining the model using historical data, with no explicit when-to-use or when-not-to-use guidance. However, given that all sibling tools have distinct purposes (e.g., 'predict_pr_risk' for predictions, 'analyze_coverage' for coverage), the context is clear enough. No alternatives or exclusions are stated, so score 4.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0analyze_coveragedetect_flaky_testsget_test_historypredict_pr_riskrecord_test_runtrain_risk_modelScored across 6 tools
Each tool targets a distinct function: coverage analysis, flaky detection, risk prediction, test execution recording, history querying, and model training. There is no functional overlap, and the descriptions clearly differentiate their purposes.
Tool names consistently use a verb_noun pattern (e.g., analyze_coverage, detect_flaky_tests, train_risk_model). The pattern is clear and predictable, though 'record_test_run' and 'get_test_history' slightly shift the pattern (record vs. get, test_run vs. test_history), but it's still minor and readable.
With 6 tools, the set is well-scoped for a CI/testing intelligence server. Each tool covers a necessary stage (record, query, analyze, detect flakiness, predict risk, train model), and none feel redundant or extraneous.
The tool set covers the core lifecycle: recording test runs (record_test_run), querying history (get_test_history), analyzing coverage (analyze_coverage), detecting flakiness (detect_flaky_tests), predicting risk (predict_pr_risk), and training the risk model (train_risk_model). A minor gap is the lack of a tool to delete or manage stored history, but the essential pipeline is complete.
An MCP server that gives your AI access to the source code and docs of all public github repos
MCP server for building and testing AI agents with multi-model experimentation and insights.
Nifty's MCP server — exposes tasks, projects, messages, and files as tools for AI agents.
