siftwise
Using this MCP server, you can select the tests a change needs, triage test failures, and explain test selection or failure behavior.
select_tests— get the command and reasons for tests affected by committed and uncommitted changes, with options for base ref, repo path, and max listed tests.triage_failure— classify failures asnew_regression,pre_existing,flaky, orunknownfrom JUnit XML reports or failing test IDs/messages, optionally by variant.explain— inspect one test’s recent results, flaky or main-branch status, and why siftwise would or would not select it.All tools work with a target repository, base ref, and current changes; the MCP server keeps no state and never records runs.
Posts sticky GitHub pull request comments with shadow-run results, test-selection outcomes, failure verdicts, and any tests the selection would have missed.
Provides a GitHub Action to run siftwise test selection in CI, with inputs for risk ordering and time budgets, and integrates with GitHub Actions workflows.
Integrates with GitLab CI using SIFTWISE_ORDER and SIFTWISE_BUDGET environment variables and posts GitLab merge request notes with selection and failure information.
Supports Jest test selection and risk ordering, listing Jest files riskiest first with the caveat that Jest may reorder them when run in parallel.
Supports pytest test selection and risk ordering, returning a pytest command and accepting pytest node IDs or JUnit results for failure triage.
2,049 / 2,050 real failures caught replaying rdflib's CI (the miss: a flaky test) · /select in ~0.4 s on 18,000 tests · self-hosted, Apache-2.0
siftwise is a self-hosted service for teams whose CI is flooded with pull requests, many of them now written by coding agents. It records every CI test result, picks the tests a change can affect (running the full suite whenever it isn't sure), and tells people and agents whether each failure is a new regression, already broken on main, or flaky. What sets it apart: it runs on your own Postgres with no test instrumentation; it fails open, so a siftwise outage never blocks CI; it ships an MCP server, so coding agents ask which tests to run and why a test failed; and its shadow mode measures, on your own CI, how often a selection would have missed a real failure before you rely on it. Why now: Anthropic reports a 25x increase in CI jobs over six months as agents sped up writing code (Agentic coding is straining CI), and Linear found that as development accelerates, CI becomes a bottleneck, because every PR still has to pass through it (AI coding has made CI a bottleneck).
Status: alpha (
0.3.0). Evaluated on six real repos' CI history. APIs and the database schema may change before 1.0.
The sticky PR comment on a shadow run: what siftwise would have run, each failure with its verdict, and the failure the selection would have missed, with its cause.
How it works
flowchart LR
A["CI run, PR or<br/>agent branch"] --> B["siftwise select<br/>(git diff + import graph)"]
B --> C[("siftwise server<br/>Postgres per-test history")]
C --> D["test command<br/>+ reasons"]
D --> E["run tests"]
E --> F["record JUnit"]
F --> C
F --> G["rollup: triage verdicts<br/>+ shadow compare"]
G --> H["PR comment"]Full details: ARCHITECTURE.md.
Record. CI uploads JUnit XML with the commit, branch, changed files and what the change links to. The server keeps every result and rolls up per-test history.
Select. The CLI diffs your branch and builds Go, Python, JS/TS (Jest) or Java import graphs locally; the server returns the tests the change can affect, each with a reason, or the full suite when the diff, a build file or the history leaves any doubt. The command runs the riskiest tests first (risk order).
Triage. Each failure gets a verdict (new regression, pre-existing on main, flaky, unknown) with a confidence and its evidence, using only history from before the run.
Measure. Shadow runs run the full suite and report what the selection would have caught or missed, and why.
Risk order
The selected tests come riskiest first, so a likely failure shows up early. Each gets a score
from how directly the change reaches it, how recently and how often it failed (flaky failures
discounted), and whether it's new: a fixed, documented formula, no learning
(details and an example).
A test whose own file imports changed code (a Python, Jest or Java test file) counts as
directly reached, like a test named after the changed file; one that reaches it only through
other files counts as indirect. Go packages that import a changed package stay indirect:
Go's graph is per package, and most dependents import the changed package themselves, so
"direct" would only lift them all over [[depends]] rules' tests. The CLI sends each
import's depth (import_depths); with an older CLI or server, every import counts as
indirect. The order never changes which tests run.
Runner | What the order guarantees |
pytest | arguments run in the order given (unless a plugin such as pytest-xdist or pytest-randomly reorders them) |
Go | one |
Jest | files are listed riskiest first, but Jest runs files in parallel and its sequencer reorders them: a hint, unless you run in band with your own sequencer |
/select returns each test's score, rank and rank_reason (siftwise select --json
shows them, MCP select_tests lists tests riskiest first). The PR comment and GitLab note
name the three highest-risk tests. To keep the old order (by path and test ID):
siftwise select --order default, the Action's order: default, SIFTWISE_ORDER: default
on GitLab, or the MCP argument order: "default".
Time budget
When there is only time for part of the selection, set a budget: siftwise runs the highest-risk tests whose estimated time fits and says plainly what it left out. Off by default. A budget is an estimated target, not a hard cap: the run can take longer.
siftwise select --repo acme/shop --base origin/main --budget 5m # s, m or h: 90s, 5m, 1h30mThe Action takes budget: 5m, GitLab SIFTWISE_BUDGET: 5m, MCP select_tests a budget
argument, and /select budget_seconds.
What runs: whole pytest arguments, Go packages and Jest files, taken in risk order when their estimated time still fits (each test's mean duration from history; a test that only ever skipped counts as 0, any other without a duration as the 90th percentile of the known ones). One that doesn't fit is skipped and the next is tried. The riskiest one always runs, even if it alone is over the budget.
A target, not a cap: the estimates are mean durations, so a run can take longer than the budget. On the eval replays, substrate runs went over in up to 31 of 72 runs, by up to 1.4× at 25% and 50% budgets, and by up to 3.9× (rdflib) at 10%, when the riskiest unit alone was over the budget. Keep a real CI timeout if you need a hard limit.
What you're told: the estimated time against the target, how many selected tests were left out, their estimated time, and their share of the selection's risk score. The PR comment and GitLab note say budget-limited (estimated 4m 50s of a 5m target): N tests left out, the CLI prints the same on stderr, and MCP returns it with
budget_limited. A budget-limited run is not a complete check of the change: run the full selection before merging.Full-suite fallbacks: when siftwise would run the full suite (unknown diff, build file changed, ...), a budget packs the full suite's known tests by risk and labels the run a budget-limited full suite. With no history to rank by, the budget isn't applied and the full suite runs.
Shadow mode: misses the budget left out are counted, with the cause
budget, and shown apart insiftwise report.An older server rejects the budget (its request schema doesn't know
budget_seconds): the Action, GitLab and MCP then run the full suite with a warning that the server doesn't support budgets;siftwise selectexits 1 with that error. Upgrade the server to use it.
How much a budget catches, on real history: eval results.
Related MCP server: knowing
Quickstart
From zero to a first selection in about 5 minutes, on a small example project. You need Docker, git and uv. The commands are bash: on Windows, use Git Bash or WSL. CI runs the blocks below on every change (quickstart.yml), with the server image and the CLI built from the checkout.
git clone https://github.com/PramodBasavarajMenasi/siftwise.git
cd siftwise1. Start the server with Postgres, from the published image
(ghcr.io/pramodbasavarajmenasi/siftwise:0.3.0). The API token is a shared secret you choose;
it goes in .env, which docker compose reads:
[ -f .env ] || echo "SIFTWISE_API_TOKEN=$(openssl rand -hex 32)" > .env
docker compose up -d --wait # pulls the image, waits until both services are healthy
curl -fsS http://localhost:8000/healthz # {"status":"ok"}To run the server from your checkout instead (when you change it), add --build.
2. Install the CLI (the light install, without the server's dependencies) and give it the token:
uv tool install siftwise # or: pip install siftwise
export SIFTWISE_API_TOKEN=$(sed -n 's/^SIFTWISE_API_TOKEN=//p' .env)If siftwise isn't found afterwards, run uv tool update-shell once and open a new shell.
3. Record a first run. Copy the example project into a git repo of its own, run its tests with a JUnit report, and upload the report:
demo="$(mktemp -d)/shop" && cp -r examples/quickstart "$demo" && cd "$demo"
git init -q && git add -A && git commit -qm "A shop"
uv run pytest -q --junitxml=junit.xml
siftwise record junit.xml --repo demo/shop --branch main --main4. Change a file and ask what to run. shop/cart.py changes; only the cart's tests import
it:
echo "# reviewed the bulk discount" >> shop/cart.py && git commit -qam "Touch the cart"
siftwise select --repo demo/shop --base HEAD~1 > selected.sh
cat selected.sh # pytest tests/test_cart.py
uv run sh selected.shThe command goes to stdout (so eval "$(siftwise select ...)" works, or a file as here) and a
summary to stderr:
siftwise: python imports: 1 test file(s) import changed modules
siftwise: selective (2 of 4 known tests): 2 of 4 known tests selectedA change to shop/prices.py selects both test files instead (the cart imports prices).
--json prints the full response with a reason per test. Exit codes: 0 ok, 1 API error, 2
usage error, 3 full suite needed but no command known. docker compose down stops the server
(down -v also deletes its data).
On your own project: record each CI run's JUnit report (the GitHub Action
does it for you, or siftwise record below), then call siftwise select (or the Action's
select step) on a change. In CI, check out with fetch-depth: 0 so the base...head diff works;
if it fails, the full suite runs.
Install
Install | What you get | Size |
| the CLI: | 18 MB |
| plus the API server: | 50 MB |
| plus | 50 MB |
For a command-line tool, uv tool install siftwise (or pipx install siftwise) gives it its
own environment; add the extras the same way (uv tool install 'siftwise[mcp]'). From a
checkout: uv tool install . (or ".[server]", ".[mcp]"). A command whose extra is missing
exits 2 and prints the install command. siftwise --version prints the version.
Run the server
With Docker, from the published image:
docker run -d -p 8000:8000 -e SIFTWISE_API_TOKEN=... \
-e SIFTWISE_DATABASE_URL=postgresql+psycopg://user:password@db-host:5432/siftwise \
ghcr.io/pramodbasavarajmenasi/siftwise:0.3.0To build the image from a checkout instead: docker build -t siftwise ., then run siftwise
in place of the image name.
The image (about 80 MB to pull, 350 MB unpacked) runs
siftwise server startas a non-root user: it applies the database migrations, then serves on port 8000. Migrations take a Postgres lock, so several replicas can start at once. A Docker health check calls/healthz.Without Docker:
pip install 'siftwise[server]', thensiftwise server start --host 0.0.0.0(siftwise server migrateapplies the migrations alone).Reaching it from CI: the quickstart's
docker-compose.ymlpublishes the API on127.0.0.1only (portSIFTWISE_PORT, default 8000) and keeps Postgres inside Docker. Put the server behind a reverse proxy with TLS: the token travels in a request header.Upgrading: change the image tag to the new version (the quickstart's
docker-compose.ymlnames it; a newer checkout already has it), thendocker compose up -d: the new image migrates the database when it starts. Never go back to an older image: once the database has a migration that image doesn't know, itssiftwise server startfails withCan't locate revision ...and the container restarts in a loop. If you build from a checkout (--build), rebuild after every pull for the same reason. A few migrations need a step after them: see the migration notes.
Seed history from past CI runs
Backfill a repo's past GitHub Actions runs (here rdflib, which uploads one JUnit artifact per
matrix leg) instead of waiting for new ones. GITHUB_TOKEN needs actions:read and
contents:read:
export GITHUB_TOKEN=...
uv run python scripts/backfill.py --repo RDFLib/rdflib --workflow validate.yaml \
--artifact-pattern '*-pytest-junit-xml' --max-runs 20 --batchRecord results from other CI systems
On GitHub, use the GitHub Action; on GitLab, the CI template. Elsewhere, upload JUnit XML after each test run:
siftwise record junit.xml --repo acme/shop --branch feature-x --ci-run-id 1234 \
--base origin/main --started-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"Uploads are idempotent per (repo, ci_run_id, run_attempt, variant): a re-upload stores
nothing new. For a matrix, give each leg a --variant (e.g. ubuntu-py3.12) and
--defer-rollup, then run siftwise rollup --repo acme/shop once after all legs. With
--base, record also stores what the change links to, which triage needs to tell whether a
later failure was related to this run's change.
Triage from the CLI
siftwise triage reads a run's JUnit reports and its diff, and gives each failing test a
verdict:
siftwise triage junit.xml --repo acme/shop --base origin/main --variant ubuntu-py3.12 \
--ci-run-id 1234 --fail-on new_regressionnew_regression high tests.test_cart::test_total
tests/test_cart.py tests changed src/cart.py
pre_existing high tests.test_db::test_query
failing on main on ubuntu-py3.12 since 4f2a9c1 (3 main runs)
flaky medium tests.test_net::test_fetch
flaky 1 time in 90 days (1 unrelated fail-then-pass)
siftwise: 3 failing test(s): 1 new regression, 1 pre existing, 1 flaky, 0 unknownThe rules, in order, are in the architecture doc.
Only runs before the triaged one count, so triaging an old run gives the answer it would have
got live. --json prints the verdicts with their evidence. Exit codes: 0 ok, 1 API error, 2
usage error, 4 a failure got a --fail-on verdict (off by default). 500 failures on rdflib
take about 0.25 s.
GitHub Action
Three composite actions in this repo. They install the light CLI from the action's own
checkout and never block CI: if the server is unreachable, times out or returns an error,
select outputs mode: full and record/rollup log a warning. The only exception is
fail-on, which you opt into.
Action | When | What |
| before tests | picks tests, riskiest first ( |
| after tests, | uploads JUnit with the run ID and attempt, the PR head commit (not the merge commit), branch, variant, start time, changed files and |
| once after a matrix | triages the failures of every leg and reports a shadow run, then updates test stats once for all legs |
Setup:
Secrets:
SIFTWISE_URL(your server) andSIFTWISE_TOKEN(itsSIFTWISE_API_TOKEN).Permissions:
contents: read,actions: read(the run's start time) andpull-requests: write(the comment).Full history: check out with
fetch-depth: 0.selectfails with a clear message on a shallow clone, the only way it fails.Version: the examples use
@main; pin a release tag (@v0.3.0) or a commit SHA for reproducible CI. The actions install the CLI from their own checkout, so the CLI's version always matches the tag.
Never use these actions with pull_request_target. That event runs with secrets on code
from the PR, so a fork could read your siftwise token. Use pull_request: on fork PRs, secrets
are empty, so select runs the full suite and skips the comment, record skips the upload,
and there are no failure verdicts.
Single job
name: CI
on:
push:
branches: [main]
pull_request:
permissions:
contents: read
actions: read
pull-requests: write
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install -r requirements.txt # your install step
- id: siftwise
uses: PramodBasavarajMenasi/siftwise/actions/select@main
with:
server-url: ${{ secrets.SIFTWISE_URL }}
token: ${{ secrets.SIFTWISE_TOKEN }}
- name: Tests
env:
MODE: ${{ steps.siftwise.outputs.mode }}
COMMAND: ${{ steps.siftwise.outputs.command }}
JUNIT_DIR: ${{ steps.siftwise.outputs.junit-dir }}
run: |
if [ "$MODE" = selective ]; then
# Shell-quoted by siftwise; each pytest command writes its own report to JUNIT_DIR.
if [ -n "$COMMAND" ]; then eval "$COMMAND"; else echo "No tests affected."; fi
else
pytest --junitxml="$JUNIT_DIR/full.xml" # your normal full suite
fi
- if: always()
uses: PramodBasavarajMenasi/siftwise/actions/record@main
with:
server-url: ${{ secrets.SIFTWISE_URL }}
token: ${{ secrets.SIFTWISE_TOKEN }}Pass the command through env: as above, never as ${{ }} inside run:, so the shell sees
it once, quoted. record uploads **/junit*.xml plus everything in the select step's
junit-dir (its junit input takes other globs). For Go, write JUnit with
gotestsum --junitfile or go-junit-report.
Matrix
Each leg records its own variant and skips the per-upload stats rollup; one rollup job runs
after every leg:
jobs:
test:
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
python: ["3.11", "3.12"]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
# ... set up Python and install, as above ...
- id: siftwise
uses: PramodBasavarajMenasi/siftwise/actions/select@main
with:
server-url: ${{ secrets.SIFTWISE_URL }}
token: ${{ secrets.SIFTWISE_TOKEN }}
comment: ${{ strategy.job-index == 0 }} # one leg comments, or they race
# ... the Tests step, as above ...
- if: always()
uses: PramodBasavarajMenasi/siftwise/actions/record@main
with:
server-url: ${{ secrets.SIFTWISE_URL }}
token: ${{ secrets.SIFTWISE_TOKEN }}
variant: ${{ matrix.os }}-py${{ matrix.python }}
siftwise-rollup:
needs: test
if: always()
runs-on: ubuntu-latest
steps:
- uses: PramodBasavarajMenasi/siftwise/actions/rollup@main
with:
server-url: ${{ secrets.SIFTWISE_URL }}
token: ${{ secrets.SIFTWISE_TOKEN }}The rollup job needs no checkout: the server already has what each leg recorded.
Failure verdicts
After the run's last upload (rollup after a matrix, record in a single job: its triage
input, auto, true or false), siftwise triages the failing tests from what the uploads
stored, including the change links record computed. The sticky PR comment gets a
Failures section under the selection:
a summary line, e.g. "2 new regressions, 1 unknown, 1 flaky, 1 pre-existing", counting each test once, under its worst verdict over the legs;
a table of test, verdict, confidence, legs and one reason: new regressions first, then unknown; flaky and pre-existing collapsed below.
allmeans every recorded leg got that verdict;at most 20 rows per table, then "+N more"; the job summary has the full list (up to 500).
The comment is left alone if the PR got new commits since the run started, and a new push's
selection drops the old commit's failures. Raw failure messages are never shown (the comment
is public). Outputs: verdicts-available, counts (JSON, per verdict) and report (the
summary's Markdown, as a file). How each verdict is decided:
triage verdicts.
fail-on
fail-on (on rollup, or record in a single job) fails the step when a failure gets one of
the listed verdicts, on any leg:
- uses: PramodBasavarajMenasi/siftwise/actions/rollup@main
with:
server-url: ${{ secrets.SIFTWISE_URL }}
token: ${{ secrets.SIFTWISE_TOKEN }}
fail-on: new_regression,unknownIt can only add a failure. A leg whose tests failed is already red. To let siftwise decide, give the test steps
continue-on-error: trueand make the rollup job the required check.Gate on
new_regression,unknown.new_regressionalone letsunknownthrough, andunknownis common (a new test, an unknown diff, an unrelated failure with no flaky history).With
fail-onset, unavailable verdicts fail the step too (fail closed): a server that's down or erroring, a timeout, or nothing recorded for the attempt. A real failure can't pass while siftwise is unreachable.Except without a siftwise token: fork PRs get no secrets, so the step warns
fail-on skipped: no siftwise token (fork PR?); this run is not gatedand passes. Keep another required check on fork PRs. A missingSIFTWISE_TOKENsecret on your own repo looks the same: watch for that warning.A PR can't excuse its own failures with
known_flaky:recordapplies the PR base commit's list (Flaky tests).
Without fail-on (the default), siftwise never fails anything.
Flaky tests
Two ways to keep known flakes from reading as regressions, both in record:
- if: always()
uses: PramodBasavarajMenasi/siftwise/actions/record@main
with:
server-url: ${{ secrets.SIFTWISE_URL }}
token: ${{ secrets.SIFTWISE_TOKEN }}
rerun-unknown: 2 # re-run unknown failures up to twice
rerun-prefix: uv run # how pytest or Jest runs in your job (poetry run, npx)known_flakyin.siftwise.toml: a failing listed test not linked to the change is flaky (medium); linked, it's unknown ("re-run to tell"). For a pull request,recordapplies the base commit's list (known-flaky-source: base, the default), so a PR can't mark its own failures flaky; when the PR changes the list, the comment says "known_flaky changes in this PR apply after merge".known-flaky-source: headapplies the PR's own list instead. A push always applies its own.rerun-unknown: N(0 to 3, default 0): after uploading,recordtriages its own leg, re-runs failures whose verdict is inrerun-verdicts(unknown, orunknown,new_regression) with the test's own runner, and records each re-run (all of a round's reports in one upload). A test that passes on one is flaky, high; one that fails again keeps its verdict, with a note.Runners, chosen per failing test from its ID and file and the checkout:
Runner
When
Re-run command
Report
pytest
a test ID that maps to a file in the checkout
[prefix] pytest -p no:cacheprovider <node IDs>--junitxmlin the runner's temp dirGo
an import path in one of the checkout's modules (the root's
go.mod, ago.modbelow it,go.work's)per package, in its module's directory:
gotestsum ... -- -count=1 -run '^(TestA|TestB)$' <package>, orgo test -jsonwith the same flags when gotestsum isn't on PATHgotestsum's JUnit, or
go test -jsonconverted by siftwiseJest
a JS/TS test file under a
package.jsonorjest.config.*with Jest: the one the JUnit names (jest-junit'saddFileAttributeorusePathForSuiteName), else the one test file in the checkout whose string literals make up every test name of the report's suiteper file, from that package:
[prefix] jest --ci --runTestsByPath <file> -t '^(<full name>|...)$'jest-junit (it must be installed, as for the CI run)
Maven
a Java class and method,
pom.xmlat the rootmvn -B -Dtest=Class#m1+m2,Other#m3 test(./mvnwwhen present)the
target/surefire-reportsfiles the re-run wrote, copied out (then put back as they were)Gradle
a Java class and method,
build.gradle[.kts]orsettings.gradle[.kts]at the rootgradle test --continue --tests Class.method ...(./gradlewwhen present)the
build/test-resultsfiles the re-run wrote, copied out (then put back as they were)JUnit 5 display names (Gradle's reports name tests by them; so does Surefire's phrased reporter, for classes too) re-run by the method that carries them, read from the class's source: a method's
@DisplayName, or a@ParameterizedTest/@RepeatedTestinvocation that its name template (or JUnit's default) produces, which re-runs every case of the method. A name more than one method may carry (two equal display names, a display name that isn't a string literal, a@DisplayNameGeneration, a@TestFactory) isn't re-run, with the reason, and keeps its verdict.Go subtests re-run through their top-level test; Java parameter lists and
[n]indexes are dropped (all of a method's cases run). Names that reach a regex are escaped; Java class and method names must be plain identifiers, else the test isn't re-run. A test whose runner can't be told isn't re-run, with a warning.rerun-prefixgoes in front of the pytest and Jest commands, split like a shell would, e.g.uv run,poetry run,npx,yarn. Go, Maven and Gradle never get it (a prefix for one language would break another in a mixed repo): siftwise uses gotestsum if it's on PATH (go install gotest.tools/gotestsum@latestbeforerecord), elsego test -json, and the repo's./mvnwor./gradlewif present, elsemvnorgradle.Limits: at most
rerun-max-teststests (20) withinrerun-timeout-minutes(10) for all rounds and runners. Tests whose file isn't in the checkout aren't re-run. Every command runs without a shell. A re-run's report must name the test as the CI run did: with custom jest-junit name templates (or settings only the test step had), the re-run may not match, and the test keeps its verdict. Without jest-junit'saddFileAttribute, a Jest suite whose test names two test files could both make (or none) isn't re-run, with a warning: never a guess.Maven and Gradle reports: both write a re-run's reports where the CI run's are (Surefire can't be pointed elsewhere from the command line, and Gradle empties its results directory, leaving only the re-run classes'). So
recordbacks up everyTEST-*.xmlundertarget/surefire-reportsandbuild/test-resultsbefore a re-run, copies the re-run's out, and puts the CI run's back afterwards, also when the re-run fails or times out (a file it can't put back is a warning). Other output isn't restored: Surefire's.txtsummaries, Gradle's HTML report (build/reports/tests) and binary results, compiled classes. A Gradle daemon can keep writing after a timeout. A build that puts its reports elsewhere (its ownreportsDirectory) gets no re-runs: no report is found, and the re-runs stop with a warning.Fail open: any problem is a warning and the verdicts stay what they were. Re-runs never turn a red job green; the PR comment and
fail-onuse the verdicts after them. Outputs:rerun-tests,rerun-passed.
Shadow mode
How often would siftwise's selection have missed a real failure? On a sampled run the full suite runs while siftwise stores what it would have selected; after the run is recorded, the server checks every failure against that selection:
- id: siftwise
uses: PramodBasavarajMenasi/siftwise/actions/select@main
with:
server-url: ${{ secrets.SIFTWISE_URL }}
token: ${{ secrets.SIFTWISE_TOKEN }}
shadow: 10% # or always; off by defaultSampling:
off(default),always, or a percentage of CI runs, by a hash of the run ID, so every matrix leg and re-run attempt agrees. Matrix legs can setvariantonselect(the same value asrecord's) to store a selection per leg.A shadow run always runs the full suite and records its results as usual; shadow mode never fails or shortens a run.
The report: where verdicts are computed, the PR comment and job summary get a line like "Shadow run: would have run 197 of 900 tests (56% time saved) and caught 1 of 2 failures", then each missed test with its cause (see the screenshot above). Time saved is measured on the run's own durations. Failures triage calls flaky are counted apart, never as misses.
Not counted, with a warning: a selection the server couldn't store, siftwise unreachable at select time, or a comparison that failed.
Recall over time comes from GET /repos/{repo}/recall?days=30, or siftwise report
(example output):
siftwise report --repo acme/shop --days 30
# Recall 98.7% (148 of 150) over 40 shadow runs; selective runs only: 96.0% (48 of 50) over 25The first figure counts every shadow run, including those where siftwise would have run the
full suite anyway, so the report always shows the selective-only figure next to it. Below come
the misses by verdict and cause, the measured time saved, the runs not counted, and each miss.
--by week prints the same per week:
Week of Runs Recall (all) Selective Recall (selective) Caught Missed Time saved
2026-09-28 11 100.0% (41 of 41) 7 100.0% (13 of 13) 41 0 38%
2026-10-05 12 98.0% (49 of 50) 8 95.0% (19 of 20) 49 1 41%Each miss gets a cause, from evidence the server already has:
Cause | Evidence |
| a time budget left out a test the selection had (counted in recall, and shown apart) |
| no result for the test before the run |
| flaky before, but not called flaky this time |
| a non-code file (data, fixture, golden file) changed in the test's own directory, which no import graph follows; the report suggests a |
| the change is known and nothing in it links to the test (the report lists what changed) |
| the run's change links weren't recorded, or the test was linked but not selected (the report warns: possibly a selector bug) |
action-e2e.yml runs all three actions against a real server
on every push and PR of this repo: verdicts, fail-on, a shadow run with a miss,
known_flaky and rerun-unknown.
GitLab CI
A CI template, gitlab/siftwise.gitlab-ci.yml,
with three hidden jobs around siftwise gitlab select|record|rollup. They run the same steps
as the GitHub Action, with the same rules: they fail open, use the merge request's base for
known_flaky, support shadow mode and rerun-unknown, and fail only through fail-on.
Your job |
| What it does |
|
| picks the tests; passes |
your test jobs |
| its |
|
| triages every test job's failures into the note, applies |
Each job writes its reports under .siftwise-ci/ (artifacts, kept 1 week): select.md,
failures.md, and the dotenv files. GitLab has no job summary, so these files are where to
look.
Setup:
Include the template by a release tag and its hash, never a branch. The tag fixes what runs in your pipelines with your tokens; GitLab rejects the file if it changes (
include:integrity, GitLab 17.9+). Get the hash withcurl -sL <url> | openssl dgst -sha256 -binary | openssl base64 -A. Each template installs its own release (pip install siftwise==<version>). Or copy the file into your repository andinclude: local:it.CI/CD variables (Settings > CI/CD > Variables):
SIFTWISE_SERVER(your server, https),SIFTWISE_TOKEN(itsSIFTWISE_API_TOKEN; Masked and hidden) and, for the note,SIFTWISE_GITLAB_TOKEN(Masked and hidden). Choosing between protected and not: protected or unprotected.Full history: the select and test jobs set
GIT_DEPTH: "0"; new GitLab projects clone 20 commits. On a shallow clone,selectfails with that fix and the pipeline runs the full suite (the select job hasallow_failure: true).Images: the select job uses
python:3.12-bookworm, which has git. For Go import graphs, give it an image with Go and Python 3.11+. The record step runs in your test job's image, which needs Python 3.11+ (python3 -m venv).
include:
- remote: https://raw.githubusercontent.com/PramodBasavarajMenasi/siftwise/v0.3.0/gitlab/siftwise.gitlab-ci.yml
integrity: sha256-<base64 SHA-256 of that file>
workflow: # merge request pipelines for open MRs, branch pipelines otherwise
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
siftwise-select:
extends: .siftwise-select
test:
extends: .siftwise-record
image: python:3.12
script:
- pip install -r requirements.txt
- |
if [ "$SIFTWISE_MODE" = selective ]; then
# Shell-quoted by siftwise; each pytest command writes its report to .siftwise-ci/junit.
if [ -s "$SIFTWISE_COMMAND_FILE" ]; then sh "$SIFTWISE_COMMAND_FILE"; else echo "No tests affected."; fi
else
pytest --junitxml=.siftwise-ci/junit/full.xml # your full suite
fi
artifacts:
reports:
junit: .siftwise-ci/junit/*.xml
siftwise-rollup:
extends: .siftwise-rollup
variables:
SIFTWISE_FAIL_ON: new_regression,unknown # optional: see fail-onThe Tests tab: GitLab's pipeline Tests tab and the merge request's test summary show only
the reports a job declares under artifacts:reports:junit (unit test
reports), never what siftwise
uploads to its own server. .siftwise-record declares .siftwise-ci/junit/*.xml: siftwise's
pytest commands write their reports there, so write your full suite's report there too (as
above). Go, Jest, Maven and Gradle commands don't take a report path from siftwise: point
their JUnit output at .siftwise-ci/junit/ (gotestsum --junitfile, jest-junit's
JEST_JUNIT_OUTPUT_DIR), or declare their own report paths in your job (a job's
artifacts:reports:junit replaces the template's). Reports written in after_script are
uploaded too: it runs before the artifact upload.
If siftwise-select failed or didn't run, SIFTWISE_MODE is empty and the test job runs the
full suite. Test jobs that use needs: must list siftwise-select to get its variables. If
your test job has its own after_script or artifact paths, add
!reference [.siftwise-record, after_script] and .siftwise-ci/ to them.
Matrix: give each leg a SIFTWISE_VARIANT. The template's record step defers the stats
and the verdicts to siftwise-rollup, which runs once after every leg:
test:
extends: .siftwise-record
image: python:$PY
parallel:
matrix:
- PY: ["3.11", "3.12"]
variables:
SIFTWISE_VARIANT: py$PY
script:
- ... # as aboveInputs are the GitHub Action's, as variables: SIFTWISE_REPO, SIFTWISE_BASE_REF,
SIFTWISE_SHADOW, SIFTWISE_ORDER, SIFTWISE_BUDGET, SIFTWISE_COMMENT, SIFTWISE_VARIANT, SIFTWISE_JUNIT_GLOB,
SIFTWISE_FAIL_ON, SIFTWISE_KNOWN_FLAKY_SOURCE, the SIFTWISE_RERUN_* ones. The rollup's
outputs are SIFTWISE_VERDICTS_AVAILABLE, SIFTWISE_COUNT_<VERDICT>, SIFTWISE_REPORT and
the shadow ones.
Try it on gitlab.com with examples/gitlab: a small project whose
pipeline runs its own siftwise server and Postgres inside each job, with a random token, so
the only variable it needs is SIFTWISE_GITLAB_TOKEN. With no server between pipelines, it
keeps history only within a pipeline: a merge request pipeline first runs the full suite at
the MR's base commit and records it as a main run. A real setup runs one persistent server
and needs neither.
The GitLab token
The note goes through the Notes API. Give it the least that works:
Premium or Ultimate, or Self-Managed: a project access token with the Reporter role, the
apiscope (read_apican't write notes) and an expiry date. Its bot user writes the note. Planner is the lowest role that can comment, but not when merge requests are set to "Only Project Members" (permissions).GitLab.com Free (no project access tokens): a dedicated user (a bot account, a project member with the Reporter role) with a fine-grained personal access token limited to this project: Work Item create, read and update (merge request notes are part of it), Merge Request read, and User read (siftwise looks up its own user so it only edits its own note). Set an expiry date. A classic token with the
apiscope also works, but it can do anything its user can, everywhere.CI_JOB_TOKENcan't do it: its allowlist has onlyGETendpoints for merge requests and notes (job token), so it can read notes but not write them.
The role should stay below Maintainer: Maintainers can edit other users' comments. A token
without access just warns (merge request comment failed: HTTP 403), and the pipeline goes on.
Protected or unprotected variables
Protected variables reach only pipelines on protected branches and tags. A merge request from an unprotected branch (most of them) gets no token: siftwise runs the full suite, posts no note and records nothing. With
fail-onset,siftwise-rollupthen fails closed: only forks skip the gate, so don't combine protected variables andfail-on. GitLab 18.1+ can pass protected variables to merge request pipelines, but only when both the source and target branches are protected (details).Unprotected (masked and hidden) variables reach every branch pipeline. Anyone who can push a branch (the Developer role) can change
.gitlab-ci.ymlon it and read both tokens. Masking hides a value in logs, not from the job's code.Recommended: unprotected, masked and hidden, when everyone with the Developer role may read and write siftwise history. The siftwise token reads and writes every repo on its server: run a separate siftwise server per trust boundary, so that a leaked token exposes only one team's repos. Otherwise use protected variables: siftwise then helps only protected branches.
Forks
By default a fork's merge request pipeline runs in the fork, without this project's
variables: the full suite, no note, no upload, and fail-on skipped with a warning (as for
fork PRs on GitHub). Keep another required check on them. Never run untrusted forks'
pipelines in the parent project (Run pipeline on the merge request's Pipelines tab): that
runs the fork's code with this project's variables, so it could read both tokens. siftwise
warns when it sees one. Like GitHub's pull_request_target, only do it for code you've read.
GitLab security
Tokens come from the environment only. No
--tokenoption exists, and siftwise redacts every value ofSIFTWISE_TOKEN,SIFTWISE_API_TOKEN,SIFTWISE_GITLAB_TOKENandCI_JOB_TOKENfrom its log lines, dotenv files, reports and the note, even when an error message echoes one. Never addset -xorCI_DEBUG_TRACEto these jobs.Untrusted text is inert in the note. Test IDs, file paths, reasons and variant names come from the merge request's code, so siftwise writes them as code spans, which GitLab shows literally: no links, images, HTML,
@mentions, references or placeholders. No line can start with/, so none can run a quick action. Raw failure messages are never shown.A merge request can't excuse its own failures:
recordapplies theknown_flakylist of the MR's base commit.Plain http warns:
SIFTWISE_SERVERwithhttp://and a host other thanlocalhostor a one-word service name (the token travels in a header) gives a warning.Protect
.gitlab-ci.yml: a merge request that edits it can change any gate. Protect the default branch and, with Premium or Ultimate, require code owner approval for the file (CODEOWNERS).
Limits:
Retried jobs: a retried job keeps its pipeline ID, so only the pipeline's first upload of each leg is stored (
recordwarns). Run a new pipeline to record again.No diff base: merge trains, scheduled, manual (web), API and trigger pipelines run the full suite.
after_scriptlimits: it runs with a 5-minute limit by default (RUNNER_AFTER_SCRIPT_TIMEOUT), which capsrerun-unknown, and, depending on the executor, without software thescriptsection installed. Re-runs need the test dependencies inside the project directory (a virtualenv there,node_modules), or callsiftwise gitlab recordat the end ofscript.Tested mostly with mocks: the GitLab support is tested against GitLab's documented variables and API, with mocks, plus the example project on gitlab.com (Free tier). Self-Managed instances and Premium features are untested.
Coding agents (MCP)
siftwise mcp is an MCP server (stdio) that gives a coding agent three tools. It keeps no
state: each call computes the change in the agent's checkout, with the same code as
siftwise select, and asks the siftwise server. It never records runs.
Tool | The agent calls it | Returns |
| before running tests after a change |
|
| when tests fail, before trying to fix them | a verdict per failure (from JUnit paths or test IDs), with reasons and evidence |
| when a selection or verdict surprises it | one test's recent results, flaky evidence, broken-on-main status, and why it would (not) be selected |
Uncommitted work counts. The diff runs from the merge base of the base branch and HEAD to the working tree, untracked files included. When the tree is dirty, triage sends no commit, so "passed on this commit before" can't apply.
Fails open. If the server is down, errors or no token is set,
select_testssays to run the full suite andtriage_failuresays verdicts are unavailable.Compact output. Each result stays under 8,000 characters. The API token never appears in a result.
Install the CLI with the mcp extra (Install):
uv tool install 'siftwise[mcp]' # siftwise lands in ~/.local/bin (Windows: %USERPROFILE%\.local\bin)
uv tool update-shell # once, if that directory isn't on your PATH; then restart the agent
uv tool dir --bin # prints the directory, for a full path in the configs belowThe server reads SIFTWISE_SERVER and SIFTWISE_API_TOKEN, and optionally SIFTWISE_REPO
(default: owner/name from the origin remote), SIFTWISE_BASE_REF (default: the branch
origin/HEAD points to, else main) and SIFTWISE_REPO_PATH. Keep the token out of files
you commit: use your user-level config, or environment-variable expansion where the client has
it.
Which repository: the first that applies of the tool's repo_path argument (an absolute
path the agent can pass), SIFTWISE_REPO_PATH, CLAUDE_PROJECT_DIR (set by Claude Code), the
client's workspace folders if it supports MCP roots, and the server's working directory.
A folder the client sets (CLAUDE_PROJECT_DIR, a root) that isn't a repository but holds
exactly one directly inside means that one, and the result says so. Outside a git repository,
every tool says so and how to fix it, listing the repositories one level down.
Tested on Windows with Claude Code, GitHub Copilot in VS Code, Cursor, Codex CLI and Gemini CLI. Windsurf and Cline configs follow their docs but are untested. Setup for each, with snippets: MCP clients.
Client | Config | How it finds the repository |
Claude Code (tested) |
|
|
GitHub Copilot in VS Code (tested) |
| MCP roots, else the workspace folder it starts in |
Cursor (tested) |
| MCP roots: no path to configure |
Codex CLI (tested) |
| the working directory: set |
Gemini CLI (tested) |
| the working directory: set |
Windsurf (untested) |
|
|
Cline (untested) |
|
|
Claude Code:
claude mcp add siftwise --env SIFTWISE_SERVER=https://siftwise.example.com \
--env SIFTWISE_API_TOKEN=your-token -- siftwise mcpor a project .mcp.json, which expands ${VAR} from your environment:
{
"mcpServers": {
"siftwise": {
"command": "siftwise",
"args": ["mcp"],
"env": {
"SIFTWISE_SERVER": "https://siftwise.example.com",
"SIFTWISE_API_TOKEN": "${SIFTWISE_API_TOKEN}"
}
}
}
}Codex CLI (~/.codex/config.toml):
[mcp_servers.siftwise]
command = "siftwise"
args = ["mcp"]
cwd = '/path/to/your/repo' # the repository: Codex starts the server there
env = { SIFTWISE_SERVER = "https://siftwise.example.com", SIFTWISE_API_TOKEN = "your-token" }If Codex says the model isn't supported with a ChatGPT account, pick another model with
/model.
Cursor (~/.cursor/mcp.json for every project, or .cursor/mcp.json in one): the same
JSON as .mcp.json above, with the token value written in. No repository path: Cursor sends
its workspace folders as MCP roots.
Gemini CLI (.gemini/settings.json): the same JSON with "cwd": "/path/to/your/repo"
added. Put the token in the server's env block (as above): Gemini CLI strips inherited
variables whose names contain TOKEN, so one set only in your shell never reaches siftwise.
GitHub Copilot in VS Code (.vscode/mcp.json): servers go under "servers" with
"type": "stdio"; see MCP clients
for a config that prompts for the token.
If the agent can't find siftwise (it doesn't see your shell's PATH, common on Windows),
give the full path from uv tool dir --bin as the command. In JSON, double the backslashes
("C:\\Users\\you\\.local\\bin\\siftwise.exe"); in TOML, use a single-quoted string.
Then tell the agent when to use it, e.g. in AGENTS.md or CLAUDE.md:
Before running tests, call the siftwise `select_tests` tool and run the command it returns.
When tests fail, call `triage_failure` before fixing anything: fix `new_regression` failures,
re-run `flaky` ones, and leave `pre_existing` ones (already failing on main).Configuration
.siftwise.toml at the root of the repo being tested, read by the CLI, the Action and the
MCP server:
always_run = ["tests/smoke/**", "tests.test_health::*"] # test file paths or test IDs
generated = ["build/reports/**"] # more files your test runs write (see below)
known_flaky = ["tests/test_net.py"] # triage: unlinked failures are flaky
[[depends]] # dependencies no import graph can see
tests = "test/cli/**" # test file paths, or <Go package dir>/<TestName>
on = ["**/*.go", "!**/*_test.go"] # ! excludesGlobs:
**spans directories, while*and?stay within one path segment (tests/*matchestests/test_a.py, nottests/unit/test_b.py). Go test IDs contain their package's import path, so match them with**::TestNameor<import path>::*.Generated files never count as changed code, even untracked or committed by mistake:
junit*.xml,coverage.xml,.coverage,.coverage.*,htmlcov/,*.log,.pytest_cache/and__pycache__/, plus editor and coding-agent settings:.vscode/,.cursor/,.idea/,.mcp.json,.codex/and.gemini/, all at any depth.generatedadds to that list.known_flakyis for triage only (selection still runs listed tests); CI applies a pull request's base list (Flaky tests).
Variable | Used by | Default |
| server (required: if unset, every endpoint but | none |
| clients: the CLI's |
|
| MCP: the repository, when the client starts the server elsewhere (which repository) | none |
| server |
|
| server: per-test stats window (0 = all history) |
|
| server: request body limit | 50 MB |
|
| none |
Supported languages:
Language | Selection | Commands |
Python (pytest) |
|
|
Go | changed package and every package whose tests import it ( |
|
JS/TS (Jest) |
|
|
Java (Maven/Gradle) |
| per module: |
The JS/TS graph is safe where it can't see: a test that reaches a require() or import()
of a computed path runs for every JS change (but never counts as covering one), a changed
file no static import reaches still falls back to the full suite, and tsconfig, jsconfig,
Jest, Babel and SWC configs, root manifests and lockfiles force it as before. A
jest.config.js/.ts can't be read without Node: the selection's reasons say so, and
resolution uses tsconfig and relative paths. --no-js-imports turns the graph off.
A computed path with a static directory is bounded instead, like a bundler's context module:
import(`./locales/${lang}.json`), require('./plugins/' + name), import.meta.glob( './pages/*.tsx') (Vite) and require.context('./icons', true, /\.svg$/) (webpack) become
imports of every checkout file below that directory (at any depth) that matches the static
end, the glob or the regex, so their tests run only when one of those files changes. The
directory may be relative or a tsconfig paths/baseUrl alias. It stays a computed path
(today's run-for-everything) when there's no static directory, the directory is just ./ or
../, a bare package, outside the checkout or missing, a Jest moduleNameMapper matches it,
a glob is root-relative or uses braces or extglobs, nothing matches, or more than 500 files
match. Like bundlers, this assumes the computed part doesn't climb out with ... On
jestjs/jest, 2 of its 40 files with computed imports became bounded: almost all of them
require a variable.
For Java, siftwise reads pom.xml modules or settings.gradle[.kts] includes (no Maven,
Gradle or JVM runs) and each module's src/main/java and src/test/java. A change to a
module's own build file (its pom.xml or build.gradle[.kts]) runs the test classes of that
module and of every module that depends on it, transitively (Maven: a POM naming its
artifactId outside <dependencyManagement>, its parent included; Gradle: project(':x'),
the root build file's counting for every project). It runs the full suite instead when the
root build file or another build file changes (settings.gradle*, gradle.properties,
gradle/, wrappers, .mvn/, lombok.config), when the module's build file is one others
inherit from or configure through (a parent or aggregator POM, packaging pom,
<dependencyManagement>/<pluginManagement>, a Gradle build file with subprojects,
allprojects, configure(...) or the like), when the build's dependencies can't be read
(an artifactId built from a ${property}, typesafe projects.x accessors, a non-literal
project(...)), when the scope holds a module whose tests the class lists may miss (Kotlin,
Groovy or Scala tests, custom source directories), when main resources
change (any code may load them at runtime), when Kotlin, Groovy or Scala sources change, when
a changed Java file is in a module it doesn't analyse (custom source directories, annotation
processors that generate code; Lombok and checkers like ErrorProne don't count), or when it
can't read the build (include of non-literals, includeBuild, Android). Reflection can't
be seen statically (Spring component scanning, ServiceLoader, Class.forName of a computed
name): declare those dependencies with [[depends]] rules. --no-java-graph turns the graph
off.
Selected classes run in one command: Maven mvn -B -pl a,b -am test -Dtest=... (one
-Dtest list for every module of the invocation, so a module whose selected class has a
fully qualified name another module also has runs on its own: the copy would run too),
Gradle gradle :a:test --tests A :b:test --tests B (each --tests filters the task before
it). Maven then runs the modules in reactor order, not risk order.
On dropwizard's history (42 Maven modules) it caught 74 of 74 failures, and its selective
runs (22 of its 148 PR runs, 5 more since module POMs are scoped) skipped a median 90.6% of
the estimated test time, but most of its PRs bump the BOM and still run everything. On
fixture-monkey (a Gradle build) it caught 3 of 3, a small history
(eval results).
Jest selection needs each test's file. jest-junit writes it only with addFileAttribute: "true" (or usePathForSuiteName), so without it record finds it in
the checkout: jest-junit writes one <testsuite> per test file, and the one Jest test file
(under a package.json with Jest or a jest.config.*) whose string literals make up all of the suite's test names (titles joined by spaces,
test.each placeholders included) is the file, which record adds to the report it
uploads. Two files that could both be it, or none (titles built from variables or template
literals), leave the suite without a file, with a warning: its tests run with the full
suite, as before. A title made only of placeholders (test.each(...)('%s')) matches no name,
so such tests aren't found either. The search reads only files matching Jest's default
testMatch (a custom testMatch/testRegex isn't read) in a Jest project, skips
node_modules and build folders, reads them once per command, and stops after 20 seconds:
the suites not found by then get no file, with one warning. jestjs/jest (1,362 test files)
took about 3 s with a cold cache on Linux and 45 s on a cold Windows checkout. Setting
addFileAttribute: "true" avoids the search.
For Go, siftwise finds every module of the checkout: the root's go.mod, go.mod files
below it (not in vendor/, testdata/ or directories starting with . or _) and
go.work's use directories. go list runs in each (it never crosses into a nested module,
not even in a workspace), and the graphs are merged: a package of one module that another
imports through a local replace or the workspace links their tests. A package of a module
below the root runs as go test -C <module dir> ./pkg (from the root, Go can't build it).
go list runs with -mod=mod turned into -mod=readonly, so selecting tests never edits
go.mod or go.sum; -mod=vendor is kept. A GOFLAGS without a -mod setting gets
-mod=readonly too, unless the module is vendored (vendor/modules.txt).
A Jest test file new in the change (*.test.*, *.spec.* or under __tests__/) runs first,
by its path, once siftwise can tell the path Jest expects from a known Jest test in the same
directory, a parent or a matching project-relative directory. Otherwise, and for a new file
outside those patterns, the full suite runs as before. Shadow mode can't credit the tests in
a new Jest file (it doesn't know their IDs yet): a failure there counts as a new_test miss.
Results on real repos
Replays of real CI history with no peeking at the future, with the current code except where noted. How they measure, and what they don't prove: eval methodology; full numbers: eval results.
Repo | Failures caught | Test time skipped |
ipfs/kubo (Go, ~4,100 tests), measured with the week-2 code | 19 of 19 in failing PRs | 30.7% over 20 main commits replayed as PRs; 0% on recent PRs (all dependency/CI bumps) |
RDFLib/rdflib (Python, ~18,000 tests) | 2,049 of 2,050 in 39 failing PRs; 140 of 141 in selective runs. The miss is a flaky remote-service test | 10.4% on the 10 most recent PRs; 0.6% over 19 main commits |
quantumlib/Cirq (Python monorepo, 9 packages, ~40,000 tests) | not measurable: JUnit only from main, which never failed | 37.9% over 132 merged PRs; 79.8% for |
agent-substrate/substrate (Go, ~9,000 tests, 8 days of history) | 2,043 of 2,044 in 72 failing PRs; 38 of 39 in selective runs, with declared rules for its e2e and functional suites (26 of 38 without them, week-3 code) | 5.1% on PR runs and 13.9% over 40 main commits, with both rules (19.5% / 31.7% without rules, week-3 code) |
Savings follow package boundaries. In the Cirq monorepo, a change to one integration package skips most of the suite; a change to the core package everything imports skips little. rdflib's
__init__.pyand plugin registry connect nearly every module to nearly every test, so safe file-level selection still runs ~99% of its suite (its 10.4% comes from lint-only PRs). Bigger savings need codebases with independent parts, or per-test coverage.The misses: substrate's are end-to-end suites that drive a deployed system (no import edge reaches them; a declared dependency catches them at the cost of most of the savings) and flaky tests. rdflib's one miss,
test_service_with_bound_solutions, calls a remote SPARQL service and had failed in unrelated runs before: it's flaky, which is whatrerun-unknownis for.
Triage (a verdict per failure, scored against what happened next; details):
The worst error, a regression called flaky: 0 on rdflib. On substrate, 5 of 406 labelled regressions, all on tests with earlier same-commit flakes; 4 are likely label noise from merge diffs.
The prediction tests held up:
new_regression319 of 319 labelled substrate failures (and 10 of 11 rdflib fix episodes with a known diff), and medium-confidencepre_existing759 of 759, each called after a single failing main run.Flaky precision is mostly by construction: almost every
flakyverdict is a pass of the same commit seen before the failure, which is also the flaky label's evidence.
Performance (rdflib history; results identical before and after;
details): triage of a 22-variant CI run 1,032 → 220 ms
(180 → 12 queries, no longer growing with variants); the selection inside /select 605 → 237
ms; ingesting a 9,891-result upload about 2.2 → 1.25 s.
How it compares
From each project's public docs as of October 2026; corrections welcome.
Open source | Self-hosted | Languages | How it selects | Agent/MCP support | |
siftwise | Yes (Apache-2.0) | Yes | Python, Go, JS/TS | file mapping, Go/Python import graphs, failure history | MCP server (select, triage, explain) |
No | Not stated | Many (runner integrations incl. Python, Go, Java, JS) | LLM analysis of changes + test history | Not documented | |
No | No (Datadog SaaS) | JS/TS, Java, Python, Go, Swift and more | per-test code coverage | Not documented | |
No | Yes (Develocity server) | Gradle and Maven builds | predictive model from past builds | Not documented | |
Yes (MIT) | Local plugin, no server | Python (pytest) | per-test code coverage | No |
Coverage-based tools see dynamic dependencies that static graphs miss, but they need
instrumented runs. siftwise uses no instrumentation; when its graphs can't see a dependency,
the fallbacks and .siftwise.toml cover it.
Security and limits
Found a vulnerability? Report it privately, as SECURITY.md describes, not in a public issue.
One token per server, with full access.
SIFTWISE_API_TOKENis a single shared secret: every client (CI, the CLI, the MCP server) uses it, and it can read and write the history of every repo on the server. There are no per-repo, read-only or per-user tokens. Run a separate server for repos whose owners shouldn't see each other's history, and rotate the token by changing it on the server and in each secret.Keep the token out of fork PRs. Use
pull_request, neverpull_request_target: a fork's code must never run with the token. On GitLab, never run untrusted forks' merge request pipelines in the parent project (Forks).Protect your workflows. The base-branch
known_flakyrule assumes the workflow itself is protected. A PR that edits.github/workflows/(or.gitlab-ci.yml) can change any gate, so protect workflows with CODEOWNERS or a ruleset (on GitLab: protected branches, and code owner approval with Premium or Ultimate). GitLab's token choices: protected or unprotected variables.What the server stores: test IDs, outcomes, durations, failure messages (truncated to 4 KB), changed file paths, branch names and commit SHAs. Failure messages can contain whatever your tests print, so treat the database as you treat CI logs. Public PR comments show test IDs, verdicts and siftwise's own reasons, never raw failure messages.
Stats cover the window only. Per-test stats use the last
SIFTWISE_STATS_WINDOW_DAYS(90) days before the repo's newest run; older raw results stay, but no longer feed stats.Limits to know: merge-queue (
merge_group), scheduled and manual runs always run the full suite (no diff base); the Action is tested on Linux runners only, and the GitLab template with mocks and one example project on gitlab.com (Free).
Roadmap
Done (v0.1): recording and history, test selection with import graphs and safe fallbacks, the GitHub Action, failure triage with verdicts in the PR comment and
fail-on, the MCP server, the triage eval, shadow mode withsiftwise report, miss causes, flaky handling (known_flaky,rerun-unknown), performance (batched triage, faster select and ingest), packaging (a light CLI, a server image, the quickstart) and thev0.1.0release on PyPI and GHCR.Done (v0.2): MCP support for more clients (tested with Claude Code, GitHub Copilot in VS Code, Cursor, Codex CLI and Gemini CLI),
rerun-unknownfor Go, Jest, Maven and Gradle, and GitLab CI support (changelog).Done (v0.3): risk order, time budgets, Jest and Java (Maven and Gradle) selection, import depth in the risk order, module build-file scoping for Java, and live dogfooding numbers: every siftwise PR and push to main runs as a shadow run, with a server inside the CI job (dogfood.yml).
Next (v0.4): scoping test-fixture build files (today a fixture's
build.gradle.ktsortsconfig.jsonruns the full suite), and more recall numbers; a demo and the launch write-up; an arm64 server image.
Contributing
Issues and pull requests are welcome: see CONTRIBUTING.md for the dev setup, the checks and the PR checklist, and ARCHITECTURE.md for how the pieces fit. Everyone taking part follows the Code of Conduct.
License
Apache-2.0
Available Tools
3 toolsexplainA
Call this to understand one test: its recent results, whether it is flaky or broken on main, and why siftwise would or wouldn't select it for your current change. Use it when a selection or a triage verdict surprises you, before overriding it.
Inputs: test_id (required), base, variant, repo_path, +1 more. Returns: object
select_testsA
Call this before running tests after changing code. It returns the command that runs only the tests your change (committed and uncommitted) can affect, with a reason per test. Run that command instead of the full test suite. If mode is "full", run the project's full test suite; if the command is empty in selective mode, no tests are affected.
Inputs: base, max_tests, repo_path, include_uncommitted. Returns: object
triage_failureA
Call this when tests fail, before trying to fix them. For each failing test it says whether your change caused it (new_regression: fix these), it was already failing on main (pre_existing: not yours), it is flaky (flaky: re-run it, don't fix it), or siftwise can't tell (unknown). Pass the JUnit XML report paths from the test run, or the failing test IDs (pytest node IDs like tests/test_x.py::test_y work as classname::name, e.g. tests.test_x::test_y) with optional failure messages.
Inputs: base, variant, failures, repo_path, +2 more. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 3 tools
The three tools serve clearly distinct workflow stages: selecting affected tests before a run, triaging failing tests afterward, and explaining one test in detail when needed. The descriptions explicitly state when to call each, so an agent should not confuse them.
select_tests and triage_failure follow a clear verb_noun pattern, while explain is a bare verb. The deviation is minor and readable, but it prevents a perfect consistency score.
Three tools is a minimal but well-scoped set for the server's test-selection and triage purpose. Each tool earns its place, though the surface is at the low end of what feels comfortably complete.
The tools cover the core lifecycle: selecting tests, triaging failures, and explaining individual test behavior. Minor gaps exist around discovering/configuring test inventory or reporting outcomes, but the main workflow has no dead ends.
Maintenance
Related MCP Connectors
Change-aware CI validation and affected-test guidance for coding agents.
Test analytics for AI agents: test history, flaky tests, failure clusters, coverage.
AI Agent with Architectural Memory. Impact analysis (free), tests and code from the graph (pro).
Self-healing CI tests and proof of work for AI agents
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceThe 'Blast Radius' detector for AI Agents. Prevent regressions using Git history and organizational memory.7-

knowingofficial
AlicenseNot gradedqualityBmaintenanceContent-addressed code graph that produces ranked context for AI agents in one call. 22 MCP tools across indexing, blast radius, test scope, semantic diff, runtime traffic, and feedback-aware context packing. Incremental updates via Merkle DAG (no re-indexing). GCF wire format saves 84% tokens vs JSON24Apache 2.0- FlicenseAqualityDmaintenanceProvides AI coding agents with dependency analysis, impact detection, and build verification tools.14-
- AlicenseNot gradedqualityBmaintenanceMCP server for test impact analysis and code intelligence. Maps tests to code and git history to determine impacted tests, risk scores, and ownership for AI coding agents.8MIT