Skip to main content
Glama

2,049 / 2,050 real failures caught replaying rdflib's CI (the miss: a flaky test) · /select in ~0.4 s on 18,000 tests · self-hosted, Apache-2.0

siftwise is a self-hosted service for teams whose CI is flooded with pull requests, many of them now written by coding agents. It records every CI test result, picks the tests a change can affect (running the full suite whenever it isn't sure), and tells people and agents whether each failure is a new regression, already broken on main, or flaky. What sets it apart: it runs on your own Postgres with no test instrumentation; it fails open, so a siftwise outage never blocks CI; it ships an MCP server, so coding agents ask which tests to run and why a test failed; and its shadow mode measures, on your own CI, how often a selection would have missed a real failure before you rely on it. Why now: Anthropic reports a 25x increase in CI jobs over six months as agents sped up writing code (Agentic coding is straining CI), and Linear found that as development accelerates, CI becomes a bottleneck, because every PR still has to pass through it (AI coding has made CI a bottleneck).

Status: alpha (0.3.0). Evaluated on six real repos' CI history. APIs and the database schema may change before 1.0.

The sticky PR comment on a shadow run: what siftwise would have run, each failure with its verdict, and the failure the selection would have missed, with its cause.

How it works

flowchart LR
    A["CI run, PR or<br/>agent branch"] --> B["siftwise select<br/>(git diff + import graph)"]
    B --> C[("siftwise server<br/>Postgres per-test history")]
    C --> D["test command<br/>+ reasons"]
    D --> E["run tests"]
    E --> F["record JUnit"]
    F --> C
    F --> G["rollup: triage verdicts<br/>+ shadow compare"]
    G --> H["PR comment"]

Full details: ARCHITECTURE.md.

  • Record. CI uploads JUnit XML with the commit, branch, changed files and what the change links to. The server keeps every result and rolls up per-test history.

  • Select. The CLI diffs your branch and builds Go, Python, JS/TS (Jest) or Java import graphs locally; the server returns the tests the change can affect, each with a reason, or the full suite when the diff, a build file or the history leaves any doubt. The command runs the riskiest tests first (risk order).

  • Triage. Each failure gets a verdict (new regression, pre-existing on main, flaky, unknown) with a confidence and its evidence, using only history from before the run.

  • Measure. Shadow runs run the full suite and report what the selection would have caught or missed, and why.

Risk order

The selected tests come riskiest first, so a likely failure shows up early. Each gets a score from how directly the change reaches it, how recently and how often it failed (flaky failures discounted), and whether it's new: a fixed, documented formula, no learning (details and an example). A test whose own file imports changed code (a Python, Jest or Java test file) counts as directly reached, like a test named after the changed file; one that reaches it only through other files counts as indirect. Go packages that import a changed package stay indirect: Go's graph is per package, and most dependents import the changed package themselves, so "direct" would only lift them all over [[depends]] rules' tests. The CLI sends each import's depth (import_depths); with an older CLI or server, every import counts as indirect. The order never changes which tests run.

Runner

What the order guarantees

pytest

arguments run in the order given (unless a plugin such as pytest-xdist or pytest-randomly reorders them)

Go

one go test per package, joined with &&: packages run one after another, riskiest first; tests within a package in source order

Jest

files are listed riskiest first, but Jest runs files in parallel and its sequencer reorders them: a hint, unless you run in band with your own sequencer

/select returns each test's score, rank and rank_reason (siftwise select --json shows them, MCP select_tests lists tests riskiest first). The PR comment and GitLab note name the three highest-risk tests. To keep the old order (by path and test ID): siftwise select --order default, the Action's order: default, SIFTWISE_ORDER: default on GitLab, or the MCP argument order: "default".

Time budget

When there is only time for part of the selection, set a budget: siftwise runs the highest-risk tests whose estimated time fits and says plainly what it left out. Off by default. A budget is an estimated target, not a hard cap: the run can take longer.

siftwise select --repo acme/shop --base origin/main --budget 5m   # s, m or h: 90s, 5m, 1h30m

The Action takes budget: 5m, GitLab SIFTWISE_BUDGET: 5m, MCP select_tests a budget argument, and /select budget_seconds.

  • What runs: whole pytest arguments, Go packages and Jest files, taken in risk order when their estimated time still fits (each test's mean duration from history; a test that only ever skipped counts as 0, any other without a duration as the 90th percentile of the known ones). One that doesn't fit is skipped and the next is tried. The riskiest one always runs, even if it alone is over the budget.

  • A target, not a cap: the estimates are mean durations, so a run can take longer than the budget. On the eval replays, substrate runs went over in up to 31 of 72 runs, by up to 1.4× at 25% and 50% budgets, and by up to 3.9× (rdflib) at 10%, when the riskiest unit alone was over the budget. Keep a real CI timeout if you need a hard limit.

  • What you're told: the estimated time against the target, how many selected tests were left out, their estimated time, and their share of the selection's risk score. The PR comment and GitLab note say budget-limited (estimated 4m 50s of a 5m target): N tests left out, the CLI prints the same on stderr, and MCP returns it with budget_limited. A budget-limited run is not a complete check of the change: run the full selection before merging.

  • Full-suite fallbacks: when siftwise would run the full suite (unknown diff, build file changed, ...), a budget packs the full suite's known tests by risk and labels the run a budget-limited full suite. With no history to rank by, the budget isn't applied and the full suite runs.

  • Shadow mode: misses the budget left out are counted, with the cause budget, and shown apart in siftwise report.

  • An older server rejects the budget (its request schema doesn't know budget_seconds): the Action, GitLab and MCP then run the full suite with a warning that the server doesn't support budgets; siftwise select exits 1 with that error. Upgrade the server to use it.

How much a budget catches, on real history: eval results.

Related MCP server: knowing

Quickstart

From zero to a first selection in about 5 minutes, on a small example project. You need Docker, git and uv. The commands are bash: on Windows, use Git Bash or WSL. CI runs the blocks below on every change (quickstart.yml), with the server image and the CLI built from the checkout.

git clone https://github.com/PramodBasavarajMenasi/siftwise.git
cd siftwise

1. Start the server with Postgres, from the published image (ghcr.io/pramodbasavarajmenasi/siftwise:0.3.0). The API token is a shared secret you choose; it goes in .env, which docker compose reads:

[ -f .env ] || echo "SIFTWISE_API_TOKEN=$(openssl rand -hex 32)" > .env
docker compose up -d --wait             # pulls the image, waits until both services are healthy
curl -fsS http://localhost:8000/healthz # {"status":"ok"}

To run the server from your checkout instead (when you change it), add --build.

2. Install the CLI (the light install, without the server's dependencies) and give it the token:

uv tool install siftwise                # or: pip install siftwise
export SIFTWISE_API_TOKEN=$(sed -n 's/^SIFTWISE_API_TOKEN=//p' .env)

If siftwise isn't found afterwards, run uv tool update-shell once and open a new shell.

3. Record a first run. Copy the example project into a git repo of its own, run its tests with a JUnit report, and upload the report:

demo="$(mktemp -d)/shop" && cp -r examples/quickstart "$demo" && cd "$demo"
git init -q && git add -A && git commit -qm "A shop"
uv run pytest -q --junitxml=junit.xml
siftwise record junit.xml --repo demo/shop --branch main --main

4. Change a file and ask what to run. shop/cart.py changes; only the cart's tests import it:

echo "# reviewed the bulk discount" >> shop/cart.py && git commit -qam "Touch the cart"
siftwise select --repo demo/shop --base HEAD~1 > selected.sh
cat selected.sh                         # pytest tests/test_cart.py
uv run sh selected.sh

The command goes to stdout (so eval "$(siftwise select ...)" works, or a file as here) and a summary to stderr:

siftwise: python imports: 1 test file(s) import changed modules
siftwise: selective (2 of 4 known tests): 2 of 4 known tests selected

A change to shop/prices.py selects both test files instead (the cart imports prices). --json prints the full response with a reason per test. Exit codes: 0 ok, 1 API error, 2 usage error, 3 full suite needed but no command known. docker compose down stops the server (down -v also deletes its data).

On your own project: record each CI run's JUnit report (the GitHub Action does it for you, or siftwise record below), then call siftwise select (or the Action's select step) on a change. In CI, check out with fetch-depth: 0 so the base...head diff works; if it fails, the full suite runs.

Install

Install

What you get

Size

pip install siftwise

the CLI: select, record, triage, report, rollup, and the GitHub Action's steps

18 MB

pip install 'siftwise[server]'

plus the API server: siftwise server start and migrate (FastAPI, SQLAlchemy, Postgres driver)

50 MB

pip install 'siftwise[mcp]'

plus siftwise mcp, for coding agents

50 MB

For a command-line tool, uv tool install siftwise (or pipx install siftwise) gives it its own environment; add the extras the same way (uv tool install 'siftwise[mcp]'). From a checkout: uv tool install . (or ".[server]", ".[mcp]"). A command whose extra is missing exits 2 and prints the install command. siftwise --version prints the version.

Run the server

With Docker, from the published image:

docker run -d -p 8000:8000 -e SIFTWISE_API_TOKEN=... \
    -e SIFTWISE_DATABASE_URL=postgresql+psycopg://user:password@db-host:5432/siftwise \
    ghcr.io/pramodbasavarajmenasi/siftwise:0.3.0

To build the image from a checkout instead: docker build -t siftwise ., then run siftwise in place of the image name.

  • The image (about 80 MB to pull, 350 MB unpacked) runs siftwise server start as a non-root user: it applies the database migrations, then serves on port 8000. Migrations take a Postgres lock, so several replicas can start at once. A Docker health check calls /healthz.

  • Without Docker: pip install 'siftwise[server]', then siftwise server start --host 0.0.0.0 (siftwise server migrate applies the migrations alone).

  • Reaching it from CI: the quickstart's docker-compose.yml publishes the API on 127.0.0.1 only (port SIFTWISE_PORT, default 8000) and keeps Postgres inside Docker. Put the server behind a reverse proxy with TLS: the token travels in a request header.

  • Upgrading: change the image tag to the new version (the quickstart's docker-compose.yml names it; a newer checkout already has it), then docker compose up -d: the new image migrates the database when it starts. Never go back to an older image: once the database has a migration that image doesn't know, its siftwise server start fails with Can't locate revision ... and the container restarts in a loop. If you build from a checkout (--build), rebuild after every pull for the same reason. A few migrations need a step after them: see the migration notes.

Seed history from past CI runs

Backfill a repo's past GitHub Actions runs (here rdflib, which uploads one JUnit artifact per matrix leg) instead of waiting for new ones. GITHUB_TOKEN needs actions:read and contents:read:

export GITHUB_TOKEN=...
uv run python scripts/backfill.py --repo RDFLib/rdflib --workflow validate.yaml \
    --artifact-pattern '*-pytest-junit-xml' --max-runs 20 --batch

Record results from other CI systems

On GitHub, use the GitHub Action; on GitLab, the CI template. Elsewhere, upload JUnit XML after each test run:

siftwise record junit.xml --repo acme/shop --branch feature-x --ci-run-id 1234 \
    --base origin/main --started-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"

Uploads are idempotent per (repo, ci_run_id, run_attempt, variant): a re-upload stores nothing new. For a matrix, give each leg a --variant (e.g. ubuntu-py3.12) and --defer-rollup, then run siftwise rollup --repo acme/shop once after all legs. With --base, record also stores what the change links to, which triage needs to tell whether a later failure was related to this run's change.

Triage from the CLI

siftwise triage reads a run's JUnit reports and its diff, and gives each failing test a verdict:

siftwise triage junit.xml --repo acme/shop --base origin/main --variant ubuntu-py3.12 \
    --ci-run-id 1234 --fail-on new_regression
new_regression  high   tests.test_cart::test_total
    tests/test_cart.py tests changed src/cart.py
pre_existing    high   tests.test_db::test_query
    failing on main on ubuntu-py3.12 since 4f2a9c1 (3 main runs)
flaky           medium tests.test_net::test_fetch
    flaky 1 time in 90 days (1 unrelated fail-then-pass)
siftwise: 3 failing test(s): 1 new regression, 1 pre existing, 1 flaky, 0 unknown

The rules, in order, are in the architecture doc. Only runs before the triaged one count, so triaging an old run gives the answer it would have got live. --json prints the verdicts with their evidence. Exit codes: 0 ok, 1 API error, 2 usage error, 4 a failure got a --fail-on verdict (off by default). 500 failures on rdflib take about 0.25 s.

GitHub Action

Three composite actions in this repo. They install the light CLI from the action's own checkout and never block CI: if the server is unreachable, times out or returns an error, select outputs mode: full and record/rollup log a warning. The only exception is fail-on, which you opt into.

Action

When

What

actions/select

before tests

picks tests, riskiest first (order); outputs mode, command, selected, total, reason, junit-dir (and the shadow outputs); posts one sticky PR comment and a job summary

actions/record

after tests, if: always()

uploads JUnit with the run ID and attempt, the PR head commit (not the merge commit), branch, variant, start time, changed files and known_flaky; optionally re-runs unknown failures; in a single job, then triages the failures and reports a shadow run

actions/rollup

once after a matrix

triages the failures of every leg and reports a shadow run, then updates test stats once for all legs

Setup:

  • Secrets: SIFTWISE_URL (your server) and SIFTWISE_TOKEN (its SIFTWISE_API_TOKEN).

  • Permissions: contents: read, actions: read (the run's start time) and pull-requests: write (the comment).

  • Full history: check out with fetch-depth: 0. select fails with a clear message on a shallow clone, the only way it fails.

  • Version: the examples use @main; pin a release tag (@v0.3.0) or a commit SHA for reproducible CI. The actions install the CLI from their own checkout, so the CLI's version always matches the tag.

Never use these actions with pull_request_target. That event runs with secrets on code from the PR, so a fork could read your siftwise token. Use pull_request: on fork PRs, secrets are empty, so select runs the full suite and skips the comment, record skips the upload, and there are no failure verdicts.

Single job

name: CI
on:
  push:
    branches: [main]
  pull_request:
permissions:
  contents: read
  actions: read
  pull-requests: write
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt   # your install step

      - id: siftwise
        uses: PramodBasavarajMenasi/siftwise/actions/select@main
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

      - name: Tests
        env:
          MODE: ${{ steps.siftwise.outputs.mode }}
          COMMAND: ${{ steps.siftwise.outputs.command }}
          JUNIT_DIR: ${{ steps.siftwise.outputs.junit-dir }}
        run: |
          if [ "$MODE" = selective ]; then
            # Shell-quoted by siftwise; each pytest command writes its own report to JUNIT_DIR.
            if [ -n "$COMMAND" ]; then eval "$COMMAND"; else echo "No tests affected."; fi
          else
            pytest --junitxml="$JUNIT_DIR/full.xml"   # your normal full suite
          fi

      - if: always()
        uses: PramodBasavarajMenasi/siftwise/actions/record@main
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

Pass the command through env: as above, never as ${{ }} inside run:, so the shell sees it once, quoted. record uploads **/junit*.xml plus everything in the select step's junit-dir (its junit input takes other globs). For Go, write JUnit with gotestsum --junitfile or go-junit-report.

Matrix

Each leg records its own variant and skips the per-upload stats rollup; one rollup job runs after every leg:

jobs:
  test:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest]
        python: ["3.11", "3.12"]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      # ... set up Python and install, as above ...
      - id: siftwise
        uses: PramodBasavarajMenasi/siftwise/actions/select@main
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}
          comment: ${{ strategy.job-index == 0 }}   # one leg comments, or they race
      # ... the Tests step, as above ...
      - if: always()
        uses: PramodBasavarajMenasi/siftwise/actions/record@main
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}
          variant: ${{ matrix.os }}-py${{ matrix.python }}

  siftwise-rollup:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: PramodBasavarajMenasi/siftwise/actions/rollup@main
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

The rollup job needs no checkout: the server already has what each leg recorded.

Failure verdicts

After the run's last upload (rollup after a matrix, record in a single job: its triage input, auto, true or false), siftwise triages the failing tests from what the uploads stored, including the change links record computed. The sticky PR comment gets a Failures section under the selection:

  • a summary line, e.g. "2 new regressions, 1 unknown, 1 flaky, 1 pre-existing", counting each test once, under its worst verdict over the legs;

  • a table of test, verdict, confidence, legs and one reason: new regressions first, then unknown; flaky and pre-existing collapsed below. all means every recorded leg got that verdict;

  • at most 20 rows per table, then "+N more"; the job summary has the full list (up to 500).

The comment is left alone if the PR got new commits since the run started, and a new push's selection drops the old commit's failures. Raw failure messages are never shown (the comment is public). Outputs: verdicts-available, counts (JSON, per verdict) and report (the summary's Markdown, as a file). How each verdict is decided: triage verdicts.

fail-on

fail-on (on rollup, or record in a single job) fails the step when a failure gets one of the listed verdicts, on any leg:

      - uses: PramodBasavarajMenasi/siftwise/actions/rollup@main
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}
          fail-on: new_regression,unknown
  • It can only add a failure. A leg whose tests failed is already red. To let siftwise decide, give the test steps continue-on-error: true and make the rollup job the required check.

  • Gate on new_regression,unknown. new_regression alone lets unknown through, and unknown is common (a new test, an unknown diff, an unrelated failure with no flaky history).

  • With fail-on set, unavailable verdicts fail the step too (fail closed): a server that's down or erroring, a timeout, or nothing recorded for the attempt. A real failure can't pass while siftwise is unreachable.

  • Except without a siftwise token: fork PRs get no secrets, so the step warns fail-on skipped: no siftwise token (fork PR?); this run is not gated and passes. Keep another required check on fork PRs. A missing SIFTWISE_TOKEN secret on your own repo looks the same: watch for that warning.

  • A PR can't excuse its own failures with known_flaky: record applies the PR base commit's list (Flaky tests).

Without fail-on (the default), siftwise never fails anything.

Flaky tests

Two ways to keep known flakes from reading as regressions, both in record:

      - if: always()
        uses: PramodBasavarajMenasi/siftwise/actions/record@main
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}
          rerun-unknown: 2         # re-run unknown failures up to twice
          rerun-prefix: uv run     # how pytest or Jest runs in your job (poetry run, npx)
  • known_flaky in .siftwise.toml: a failing listed test not linked to the change is flaky (medium); linked, it's unknown ("re-run to tell"). For a pull request, record applies the base commit's list (known-flaky-source: base, the default), so a PR can't mark its own failures flaky; when the PR changes the list, the comment says "known_flaky changes in this PR apply after merge". known-flaky-source: head applies the PR's own list instead. A push always applies its own.

  • rerun-unknown: N (0 to 3, default 0): after uploading, record triages its own leg, re-runs failures whose verdict is in rerun-verdicts (unknown, or unknown,new_regression) with the test's own runner, and records each re-run (all of a round's reports in one upload). A test that passes on one is flaky, high; one that fails again keeps its verdict, with a note.

  • Runners, chosen per failing test from its ID and file and the checkout:

    Runner

    When

    Re-run command

    Report

    pytest

    a test ID that maps to a file in the checkout

    [prefix] pytest -p no:cacheprovider <node IDs>

    --junitxml in the runner's temp dir

    Go

    an import path in one of the checkout's modules (the root's go.mod, a go.mod below it, go.work's)

    per package, in its module's directory: gotestsum ... -- -count=1 -run '^(TestA|TestB)$' <package>, or go test -json with the same flags when gotestsum isn't on PATH

    gotestsum's JUnit, or go test -json converted by siftwise

    Jest

    a JS/TS test file under a package.json or jest.config.* with Jest: the one the JUnit names (jest-junit's addFileAttribute or usePathForSuiteName), else the one test file in the checkout whose string literals make up every test name of the report's suite

    per file, from that package: [prefix] jest --ci --runTestsByPath <file> -t '^(<full name>|...)$'

    jest-junit (it must be installed, as for the CI run)

    Maven

    a Java class and method, pom.xml at the root

    mvn -B -Dtest=Class#m1+m2,Other#m3 test (./mvnw when present)

    the target/surefire-reports files the re-run wrote, copied out (then put back as they were)

    Gradle

    a Java class and method, build.gradle[.kts] or settings.gradle[.kts] at the root

    gradle test --continue --tests Class.method ... (./gradlew when present)

    the build/test-results files the re-run wrote, copied out (then put back as they were)

    JUnit 5 display names (Gradle's reports name tests by them; so does Surefire's phrased reporter, for classes too) re-run by the method that carries them, read from the class's source: a method's @DisplayName, or a @ParameterizedTest / @RepeatedTest invocation that its name template (or JUnit's default) produces, which re-runs every case of the method. A name more than one method may carry (two equal display names, a display name that isn't a string literal, a @DisplayNameGeneration, a @TestFactory) isn't re-run, with the reason, and keeps its verdict.

    Go subtests re-run through their top-level test; Java parameter lists and [n] indexes are dropped (all of a method's cases run). Names that reach a regex are escaped; Java class and method names must be plain identifiers, else the test isn't re-run. A test whose runner can't be told isn't re-run, with a warning.

  • rerun-prefix goes in front of the pytest and Jest commands, split like a shell would, e.g. uv run, poetry run, npx, yarn. Go, Maven and Gradle never get it (a prefix for one language would break another in a mixed repo): siftwise uses gotestsum if it's on PATH (go install gotest.tools/gotestsum@latest before record), else go test -json, and the repo's ./mvnw or ./gradlew if present, else mvn or gradle.

  • Limits: at most rerun-max-tests tests (20) within rerun-timeout-minutes (10) for all rounds and runners. Tests whose file isn't in the checkout aren't re-run. Every command runs without a shell. A re-run's report must name the test as the CI run did: with custom jest-junit name templates (or settings only the test step had), the re-run may not match, and the test keeps its verdict. Without jest-junit's addFileAttribute, a Jest suite whose test names two test files could both make (or none) isn't re-run, with a warning: never a guess.

  • Maven and Gradle reports: both write a re-run's reports where the CI run's are (Surefire can't be pointed elsewhere from the command line, and Gradle empties its results directory, leaving only the re-run classes'). So record backs up every TEST-*.xml under target/surefire-reports and build/test-results before a re-run, copies the re-run's out, and puts the CI run's back afterwards, also when the re-run fails or times out (a file it can't put back is a warning). Other output isn't restored: Surefire's .txt summaries, Gradle's HTML report (build/reports/tests) and binary results, compiled classes. A Gradle daemon can keep writing after a timeout. A build that puts its reports elsewhere (its own reportsDirectory) gets no re-runs: no report is found, and the re-runs stop with a warning.

  • Fail open: any problem is a warning and the verdicts stay what they were. Re-runs never turn a red job green; the PR comment and fail-on use the verdicts after them. Outputs: rerun-tests, rerun-passed.

Shadow mode

How often would siftwise's selection have missed a real failure? On a sampled run the full suite runs while siftwise stores what it would have selected; after the run is recorded, the server checks every failure against that selection:

      - id: siftwise
        uses: PramodBasavarajMenasi/siftwise/actions/select@main
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}
          shadow: 10%          # or always; off by default
  • Sampling: off (default), always, or a percentage of CI runs, by a hash of the run ID, so every matrix leg and re-run attempt agrees. Matrix legs can set variant on select (the same value as record's) to store a selection per leg.

  • A shadow run always runs the full suite and records its results as usual; shadow mode never fails or shortens a run.

  • The report: where verdicts are computed, the PR comment and job summary get a line like "Shadow run: would have run 197 of 900 tests (56% time saved) and caught 1 of 2 failures", then each missed test with its cause (see the screenshot above). Time saved is measured on the run's own durations. Failures triage calls flaky are counted apart, never as misses.

  • Not counted, with a warning: a selection the server couldn't store, siftwise unreachable at select time, or a comparison that failed.

Recall over time comes from GET /repos/{repo}/recall?days=30, or siftwise report (example output):

siftwise report --repo acme/shop --days 30
# Recall 98.7% (148 of 150) over 40 shadow runs; selective runs only: 96.0% (48 of 50) over 25

The first figure counts every shadow run, including those where siftwise would have run the full suite anyway, so the report always shows the selective-only figure next to it. Below come the misses by verdict and cause, the measured time saved, the runs not counted, and each miss. --by week prints the same per week:

Week of     Runs  Recall (all)        Selective  Recall (selective)  Caught  Missed  Time saved
2026-09-28    11  100.0% (41 of 41)           7  100.0% (13 of 13)       41       0         38%
2026-10-05    12  98.0% (49 of 50)            8  95.0% (19 of 20)        49       1         41%

Each miss gets a cause, from evidence the server already has:

Cause

Evidence

budget

a time budget left out a test the selection had (counted in recall, and shown apart)

new_test

no result for the test before the run

flaky_history

flaky before, but not called flaky this time

config_gap

a non-code file (data, fixture, golden file) changed in the test's own directory, which no import graph follows; the report suggests a [[depends]] rule for you to review

no_link

the change is known and nothing in it links to the test (the report lists what changed)

unknown

the run's change links weren't recorded, or the test was linked but not selected (the report warns: possibly a selector bug)

action-e2e.yml runs all three actions against a real server on every push and PR of this repo: verdicts, fail-on, a shadow run with a miss, known_flaky and rerun-unknown.

GitLab CI

A CI template, gitlab/siftwise.gitlab-ci.yml, with three hidden jobs around siftwise gitlab select|record|rollup. They run the same steps as the GitHub Action, with the same rules: they fail open, use the merge request's base for known_flaky, support shadow mode and rerun-unknown, and fail only through fail-on.

Your job

extends:

What it does

siftwise-select

.siftwise-select (stage .pre)

picks the tests; passes SIFTWISE_MODE, SIFTWISE_COMMAND_FILE, SIFTWISE_SELECTED, SIFTWISE_TOTAL, SIFTWISE_REASON and SIFTWISE_JUNIT_DIR (plus the shadow outputs) to later jobs as a dotenv report; writes the selection into one sticky merge request note

your test jobs

.siftwise-record

its after_script uploads the job's JUnit (**/junit*.xml and .siftwise-ci/junit/) with the pipeline ID, the merge request's source commit (never a merged results commit), the branch, the variant, the changed files and the base commit's known_flaky; optionally re-runs unknown failures

siftwise-rollup

.siftwise-rollup (stage .post, when: always)

triages every test job's failures into the note, applies fail-on, reports a shadow run, then updates the test stats once

Each job writes its reports under .siftwise-ci/ (artifacts, kept 1 week): select.md, failures.md, and the dotenv files. GitLab has no job summary, so these files are where to look.

Setup:

  1. Include the template by a release tag and its hash, never a branch. The tag fixes what runs in your pipelines with your tokens; GitLab rejects the file if it changes (include:integrity, GitLab 17.9+). Get the hash with curl -sL <url> | openssl dgst -sha256 -binary | openssl base64 -A. Each template installs its own release (pip install siftwise==<version>). Or copy the file into your repository and include: local: it.

  2. CI/CD variables (Settings > CI/CD > Variables): SIFTWISE_SERVER (your server, https), SIFTWISE_TOKEN (its SIFTWISE_API_TOKEN; Masked and hidden) and, for the note, SIFTWISE_GITLAB_TOKEN (Masked and hidden). Choosing between protected and not: protected or unprotected.

  3. Full history: the select and test jobs set GIT_DEPTH: "0"; new GitLab projects clone 20 commits. On a shallow clone, select fails with that fix and the pipeline runs the full suite (the select job has allow_failure: true).

  4. Images: the select job uses python:3.12-bookworm, which has git. For Go import graphs, give it an image with Go and Python 3.11+. The record step runs in your test job's image, which needs Python 3.11+ (python3 -m venv).

include:
  - remote: https://raw.githubusercontent.com/PramodBasavarajMenasi/siftwise/v0.3.0/gitlab/siftwise.gitlab-ci.yml
    integrity: sha256-<base64 SHA-256 of that file>

workflow:  # merge request pipelines for open MRs, branch pipelines otherwise
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

siftwise-select:
  extends: .siftwise-select

test:
  extends: .siftwise-record
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - |
      if [ "$SIFTWISE_MODE" = selective ]; then
        # Shell-quoted by siftwise; each pytest command writes its report to .siftwise-ci/junit.
        if [ -s "$SIFTWISE_COMMAND_FILE" ]; then sh "$SIFTWISE_COMMAND_FILE"; else echo "No tests affected."; fi
      else
        pytest --junitxml=.siftwise-ci/junit/full.xml   # your full suite
      fi
  artifacts:
    reports:
      junit: .siftwise-ci/junit/*.xml

siftwise-rollup:
  extends: .siftwise-rollup
  variables:
    SIFTWISE_FAIL_ON: new_regression,unknown   # optional: see fail-on

The Tests tab: GitLab's pipeline Tests tab and the merge request's test summary show only the reports a job declares under artifacts:reports:junit (unit test reports), never what siftwise uploads to its own server. .siftwise-record declares .siftwise-ci/junit/*.xml: siftwise's pytest commands write their reports there, so write your full suite's report there too (as above). Go, Jest, Maven and Gradle commands don't take a report path from siftwise: point their JUnit output at .siftwise-ci/junit/ (gotestsum --junitfile, jest-junit's JEST_JUNIT_OUTPUT_DIR), or declare their own report paths in your job (a job's artifacts:reports:junit replaces the template's). Reports written in after_script are uploaded too: it runs before the artifact upload.

If siftwise-select failed or didn't run, SIFTWISE_MODE is empty and the test job runs the full suite. Test jobs that use needs: must list siftwise-select to get its variables. If your test job has its own after_script or artifact paths, add !reference [.siftwise-record, after_script] and .siftwise-ci/ to them.

Matrix: give each leg a SIFTWISE_VARIANT. The template's record step defers the stats and the verdicts to siftwise-rollup, which runs once after every leg:

test:
  extends: .siftwise-record
  image: python:$PY
  parallel:
    matrix:
      - PY: ["3.11", "3.12"]
  variables:
    SIFTWISE_VARIANT: py$PY
  script:
    - ...   # as above

Inputs are the GitHub Action's, as variables: SIFTWISE_REPO, SIFTWISE_BASE_REF, SIFTWISE_SHADOW, SIFTWISE_ORDER, SIFTWISE_BUDGET, SIFTWISE_COMMENT, SIFTWISE_VARIANT, SIFTWISE_JUNIT_GLOB, SIFTWISE_FAIL_ON, SIFTWISE_KNOWN_FLAKY_SOURCE, the SIFTWISE_RERUN_* ones. The rollup's outputs are SIFTWISE_VERDICTS_AVAILABLE, SIFTWISE_COUNT_<VERDICT>, SIFTWISE_REPORT and the shadow ones.

Try it on gitlab.com with examples/gitlab: a small project whose pipeline runs its own siftwise server and Postgres inside each job, with a random token, so the only variable it needs is SIFTWISE_GITLAB_TOKEN. With no server between pipelines, it keeps history only within a pipeline: a merge request pipeline first runs the full suite at the MR's base commit and records it as a main run. A real setup runs one persistent server and needs neither.

The GitLab token

The note goes through the Notes API. Give it the least that works:

  • Premium or Ultimate, or Self-Managed: a project access token with the Reporter role, the api scope (read_api can't write notes) and an expiry date. Its bot user writes the note. Planner is the lowest role that can comment, but not when merge requests are set to "Only Project Members" (permissions).

  • GitLab.com Free (no project access tokens): a dedicated user (a bot account, a project member with the Reporter role) with a fine-grained personal access token limited to this project: Work Item create, read and update (merge request notes are part of it), Merge Request read, and User read (siftwise looks up its own user so it only edits its own note). Set an expiry date. A classic token with the api scope also works, but it can do anything its user can, everywhere.

  • CI_JOB_TOKEN can't do it: its allowlist has only GET endpoints for merge requests and notes (job token), so it can read notes but not write them.

The role should stay below Maintainer: Maintainers can edit other users' comments. A token without access just warns (merge request comment failed: HTTP 403), and the pipeline goes on.

Protected or unprotected variables

  • Protected variables reach only pipelines on protected branches and tags. A merge request from an unprotected branch (most of them) gets no token: siftwise runs the full suite, posts no note and records nothing. With fail-on set, siftwise-rollup then fails closed: only forks skip the gate, so don't combine protected variables and fail-on. GitLab 18.1+ can pass protected variables to merge request pipelines, but only when both the source and target branches are protected (details).

  • Unprotected (masked and hidden) variables reach every branch pipeline. Anyone who can push a branch (the Developer role) can change .gitlab-ci.yml on it and read both tokens. Masking hides a value in logs, not from the job's code.

  • Recommended: unprotected, masked and hidden, when everyone with the Developer role may read and write siftwise history. The siftwise token reads and writes every repo on its server: run a separate siftwise server per trust boundary, so that a leaked token exposes only one team's repos. Otherwise use protected variables: siftwise then helps only protected branches.

Forks

By default a fork's merge request pipeline runs in the fork, without this project's variables: the full suite, no note, no upload, and fail-on skipped with a warning (as for fork PRs on GitHub). Keep another required check on them. Never run untrusted forks' pipelines in the parent project (Run pipeline on the merge request's Pipelines tab): that runs the fork's code with this project's variables, so it could read both tokens. siftwise warns when it sees one. Like GitHub's pull_request_target, only do it for code you've read.

GitLab security

  • Tokens come from the environment only. No --token option exists, and siftwise redacts every value of SIFTWISE_TOKEN, SIFTWISE_API_TOKEN, SIFTWISE_GITLAB_TOKEN and CI_JOB_TOKEN from its log lines, dotenv files, reports and the note, even when an error message echoes one. Never add set -x or CI_DEBUG_TRACE to these jobs.

  • Untrusted text is inert in the note. Test IDs, file paths, reasons and variant names come from the merge request's code, so siftwise writes them as code spans, which GitLab shows literally: no links, images, HTML, @ mentions, references or placeholders. No line can start with /, so none can run a quick action. Raw failure messages are never shown.

  • A merge request can't excuse its own failures: record applies the known_flaky list of the MR's base commit.

  • Plain http warns: SIFTWISE_SERVER with http:// and a host other than localhost or a one-word service name (the token travels in a header) gives a warning.

  • Protect .gitlab-ci.yml: a merge request that edits it can change any gate. Protect the default branch and, with Premium or Ultimate, require code owner approval for the file (CODEOWNERS).

Limits:

  • Retried jobs: a retried job keeps its pipeline ID, so only the pipeline's first upload of each leg is stored (record warns). Run a new pipeline to record again.

  • No diff base: merge trains, scheduled, manual (web), API and trigger pipelines run the full suite.

  • after_script limits: it runs with a 5-minute limit by default (RUNNER_AFTER_SCRIPT_TIMEOUT), which caps rerun-unknown, and, depending on the executor, without software the script section installed. Re-runs need the test dependencies inside the project directory (a virtualenv there, node_modules), or call siftwise gitlab record at the end of script.

  • Tested mostly with mocks: the GitLab support is tested against GitLab's documented variables and API, with mocks, plus the example project on gitlab.com (Free tier). Self-Managed instances and Premium features are untested.

Coding agents (MCP)

siftwise mcp is an MCP server (stdio) that gives a coding agent three tools. It keeps no state: each call computes the change in the agent's checkout, with the same code as siftwise select, and asks the siftwise server. It never records runs.

Tool

The agent calls it

Returns

select_tests

before running tests after a change

mode, the command to run (riskiest tests first), the selected tests with a reason and risk rank each

triage_failure

when tests fail, before trying to fix them

a verdict per failure (from JUnit paths or test IDs), with reasons and evidence

explain

when a selection or verdict surprises it

one test's recent results, flaky evidence, broken-on-main status, and why it would (not) be selected

  • Uncommitted work counts. The diff runs from the merge base of the base branch and HEAD to the working tree, untracked files included. When the tree is dirty, triage sends no commit, so "passed on this commit before" can't apply.

  • Fails open. If the server is down, errors or no token is set, select_tests says to run the full suite and triage_failure says verdicts are unavailable.

  • Compact output. Each result stays under 8,000 characters. The API token never appears in a result.

Install the CLI with the mcp extra (Install):

uv tool install 'siftwise[mcp]'  # siftwise lands in ~/.local/bin (Windows: %USERPROFILE%\.local\bin)
uv tool update-shell             # once, if that directory isn't on your PATH; then restart the agent
uv tool dir --bin                # prints the directory, for a full path in the configs below

The server reads SIFTWISE_SERVER and SIFTWISE_API_TOKEN, and optionally SIFTWISE_REPO (default: owner/name from the origin remote), SIFTWISE_BASE_REF (default: the branch origin/HEAD points to, else main) and SIFTWISE_REPO_PATH. Keep the token out of files you commit: use your user-level config, or environment-variable expansion where the client has it.

Which repository: the first that applies of the tool's repo_path argument (an absolute path the agent can pass), SIFTWISE_REPO_PATH, CLAUDE_PROJECT_DIR (set by Claude Code), the client's workspace folders if it supports MCP roots, and the server's working directory. A folder the client sets (CLAUDE_PROJECT_DIR, a root) that isn't a repository but holds exactly one directly inside means that one, and the result says so. Outside a git repository, every tool says so and how to fix it, listing the repositories one level down.

Tested on Windows with Claude Code, GitHub Copilot in VS Code, Cursor, Codex CLI and Gemini CLI. Windsurf and Cline configs follow their docs but are untested. Setup for each, with snippets: MCP clients.

Client

Config

How it finds the repository

Claude Code (tested)

claude mcp add, or .mcp.json

CLAUDE_PROJECT_DIR, which Claude Code sets

GitHub Copilot in VS Code (tested)

.vscode/mcp.json

MCP roots, else the workspace folder it starts in

Cursor (tested)

.cursor/mcp.json or ~/.cursor/mcp.json

MCP roots: no path to configure

Codex CLI (tested)

~/.codex/config.toml

the working directory: set cwd

Gemini CLI (tested)

.gemini/settings.json

the working directory: set cwd

Windsurf (untested)

mcp_config.json

SIFTWISE_REPO_PATH, or the repo_path argument

Cline (untested)

cline_mcp_settings.json

SIFTWISE_REPO_PATH, or the repo_path argument

Claude Code:

claude mcp add siftwise --env SIFTWISE_SERVER=https://siftwise.example.com \
    --env SIFTWISE_API_TOKEN=your-token -- siftwise mcp

or a project .mcp.json, which expands ${VAR} from your environment:

{
  "mcpServers": {
    "siftwise": {
      "command": "siftwise",
      "args": ["mcp"],
      "env": {
        "SIFTWISE_SERVER": "https://siftwise.example.com",
        "SIFTWISE_API_TOKEN": "${SIFTWISE_API_TOKEN}"
      }
    }
  }
}

Codex CLI (~/.codex/config.toml):

[mcp_servers.siftwise]
command = "siftwise"
args = ["mcp"]
cwd = '/path/to/your/repo'   # the repository: Codex starts the server there
env = { SIFTWISE_SERVER = "https://siftwise.example.com", SIFTWISE_API_TOKEN = "your-token" }

If Codex says the model isn't supported with a ChatGPT account, pick another model with /model.

Cursor (~/.cursor/mcp.json for every project, or .cursor/mcp.json in one): the same JSON as .mcp.json above, with the token value written in. No repository path: Cursor sends its workspace folders as MCP roots.

Gemini CLI (.gemini/settings.json): the same JSON with "cwd": "/path/to/your/repo" added. Put the token in the server's env block (as above): Gemini CLI strips inherited variables whose names contain TOKEN, so one set only in your shell never reaches siftwise.

GitHub Copilot in VS Code (.vscode/mcp.json): servers go under "servers" with "type": "stdio"; see MCP clients for a config that prompts for the token.

If the agent can't find siftwise (it doesn't see your shell's PATH, common on Windows), give the full path from uv tool dir --bin as the command. In JSON, double the backslashes ("C:\\Users\\you\\.local\\bin\\siftwise.exe"); in TOML, use a single-quoted string.

Then tell the agent when to use it, e.g. in AGENTS.md or CLAUDE.md:

Before running tests, call the siftwise `select_tests` tool and run the command it returns.
When tests fail, call `triage_failure` before fixing anything: fix `new_regression` failures,
re-run `flaky` ones, and leave `pre_existing` ones (already failing on main).

Configuration

.siftwise.toml at the root of the repo being tested, read by the CLI, the Action and the MCP server:

always_run = ["tests/smoke/**", "tests.test_health::*"]  # test file paths or test IDs
generated = ["build/reports/**"]          # more files your test runs write (see below)
known_flaky = ["tests/test_net.py"]       # triage: unlinked failures are flaky

[[depends]]                               # dependencies no import graph can see
tests = "test/cli/**"                     # test file paths, or <Go package dir>/<TestName>
on = ["**/*.go", "!**/*_test.go"]         # ! excludes
  • Globs: ** spans directories, while * and ? stay within one path segment (tests/* matches tests/test_a.py, not tests/unit/test_b.py). Go test IDs contain their package's import path, so match them with **::TestName or <import path>::*.

  • Generated files never count as changed code, even untracked or committed by mistake: junit*.xml, coverage.xml, .coverage, .coverage.*, htmlcov/, *.log, .pytest_cache/ and __pycache__/, plus editor and coding-agent settings: .vscode/, .cursor/, .idea/, .mcp.json, .codex/ and .gemini/, all at any depth. generated adds to that list.

  • known_flaky is for triage only (selection still runs listed tests); CI applies a pull request's base list (Flaky tests).

Variable

Used by

Default

SIFTWISE_API_TOKEN

server (required: if unset, every endpoint but /healthz returns 503) and clients

none

SIFTWISE_SERVER

clients: the CLI's --server, MCP, the Action's steps

http://localhost:8000

SIFTWISE_REPO_PATH

MCP: the repository, when the client starts the server elsewhere (which repository)

none

SIFTWISE_DATABASE_URL

server

postgresql+psycopg://siftwise:siftwise@localhost:5432/siftwise

SIFTWISE_STATS_WINDOW_DAYS

server: per-test stats window (0 = all history)

90

SIFTWISE_MAX_UPLOAD_BYTES

server: request body limit

50 MB

GITHUB_TOKEN

scripts/backfill.py

none

Supported languages:

Language

Selection

Commands

Python (pytest)

foo.py → test_foo.py; test files importing a changed module (static graph incl. parent packages, conftests, plugin strings); changed test files

pytest node IDs, whole test files, pytest --doctest-modules for doctests

Go

changed package and every package whose tests import it (go list -deps -test, in every module of the checkout: go.work, nested go.mod)

go test ./pkg (whole) or -run '^(TestA|TestB)$'; go test -C <module> ./pkg for a module below the root

JS/TS (Jest)

foo.ts → foo.test.*, foo.spec.*, __tests__/foo.*; test files importing a changed file (static graph, no Node needed: relative paths, tsconfig/jsconfig paths, workspace packages, a JSON Jest moduleNameMapper) (needs each test's file: jest-junit's addFileAttribute or usePathForSuiteName, or record finds it, see below)

jest <files>

Java (Maven/Gradle)

a/b/Foo.java → FooTest, FooTests, TestFoo, FooIT in its module (else any module); test classes that use a changed class (static graph, no JVM: imports, static and on-demand imports, same-package names, fully-qualified names), across modules; a module's tests for its src/test/resources

per module: ./mvnw -B -pl <module> -am test -Dtest=A,B (verify -Dit.test=... for *IT), ./gradlew :<module>:test --tests A

The JS/TS graph is safe where it can't see: a test that reaches a require() or import() of a computed path runs for every JS change (but never counts as covering one), a changed file no static import reaches still falls back to the full suite, and tsconfig, jsconfig, Jest, Babel and SWC configs, root manifests and lockfiles force it as before. A jest.config.js/.ts can't be read without Node: the selection's reasons say so, and resolution uses tsconfig and relative paths. --no-js-imports turns the graph off.

A computed path with a static directory is bounded instead, like a bundler's context module: import(`./locales/${lang}.json`), require('./plugins/' + name), import.meta.glob( './pages/*.tsx') (Vite) and require.context('./icons', true, /\.svg$/) (webpack) become imports of every checkout file below that directory (at any depth) that matches the static end, the glob or the regex, so their tests run only when one of those files changes. The directory may be relative or a tsconfig paths/baseUrl alias. It stays a computed path (today's run-for-everything) when there's no static directory, the directory is just ./ or ../, a bare package, outside the checkout or missing, a Jest moduleNameMapper matches it, a glob is root-relative or uses braces or extglobs, nothing matches, or more than 500 files match. Like bundlers, this assumes the computed part doesn't climb out with ... On jestjs/jest, 2 of its 40 files with computed imports became bounded: almost all of them require a variable.

For Java, siftwise reads pom.xml modules or settings.gradle[.kts] includes (no Maven, Gradle or JVM runs) and each module's src/main/java and src/test/java. A change to a module's own build file (its pom.xml or build.gradle[.kts]) runs the test classes of that module and of every module that depends on it, transitively (Maven: a POM naming its artifactId outside <dependencyManagement>, its parent included; Gradle: project(':x'), the root build file's counting for every project). It runs the full suite instead when the root build file or another build file changes (settings.gradle*, gradle.properties, gradle/, wrappers, .mvn/, lombok.config), when the module's build file is one others inherit from or configure through (a parent or aggregator POM, packaging pom, <dependencyManagement>/<pluginManagement>, a Gradle build file with subprojects, allprojects, configure(...) or the like), when the build's dependencies can't be read (an artifactId built from a ${property}, typesafe projects.x accessors, a non-literal project(...)), when the scope holds a module whose tests the class lists may miss (Kotlin, Groovy or Scala tests, custom source directories), when main resources change (any code may load them at runtime), when Kotlin, Groovy or Scala sources change, when a changed Java file is in a module it doesn't analyse (custom source directories, annotation processors that generate code; Lombok and checkers like ErrorProne don't count), or when it can't read the build (include of non-literals, includeBuild, Android). Reflection can't be seen statically (Spring component scanning, ServiceLoader, Class.forName of a computed name): declare those dependencies with [[depends]] rules. --no-java-graph turns the graph off. Selected classes run in one command: Maven mvn -B -pl a,b -am test -Dtest=... (one -Dtest list for every module of the invocation, so a module whose selected class has a fully qualified name another module also has runs on its own: the copy would run too), Gradle gradle :a:test --tests A :b:test --tests B (each --tests filters the task before it). Maven then runs the modules in reactor order, not risk order. On dropwizard's history (42 Maven modules) it caught 74 of 74 failures, and its selective runs (22 of its 148 PR runs, 5 more since module POMs are scoped) skipped a median 90.6% of the estimated test time, but most of its PRs bump the BOM and still run everything. On fixture-monkey (a Gradle build) it caught 3 of 3, a small history (eval results).

Jest selection needs each test's file. jest-junit writes it only with addFileAttribute: "true" (or usePathForSuiteName), so without it record finds it in the checkout: jest-junit writes one <testsuite> per test file, and the one Jest test file (under a package.json with Jest or a jest.config.*) whose string literals make up all of the suite's test names (titles joined by spaces, test.each placeholders included) is the file, which record adds to the report it uploads. Two files that could both be it, or none (titles built from variables or template literals), leave the suite without a file, with a warning: its tests run with the full suite, as before. A title made only of placeholders (test.each(...)('%s')) matches no name, so such tests aren't found either. The search reads only files matching Jest's default testMatch (a custom testMatch/testRegex isn't read) in a Jest project, skips node_modules and build folders, reads them once per command, and stops after 20 seconds: the suites not found by then get no file, with one warning. jestjs/jest (1,362 test files) took about 3 s with a cold cache on Linux and 45 s on a cold Windows checkout. Setting addFileAttribute: "true" avoids the search.

For Go, siftwise finds every module of the checkout: the root's go.mod, go.mod files below it (not in vendor/, testdata/ or directories starting with . or _) and go.work's use directories. go list runs in each (it never crosses into a nested module, not even in a workspace), and the graphs are merged: a package of one module that another imports through a local replace or the workspace links their tests. A package of a module below the root runs as go test -C <module dir> ./pkg (from the root, Go can't build it). go list runs with -mod=mod turned into -mod=readonly, so selecting tests never edits go.mod or go.sum; -mod=vendor is kept. A GOFLAGS without a -mod setting gets -mod=readonly too, unless the module is vendored (vendor/modules.txt).

A Jest test file new in the change (*.test.*, *.spec.* or under __tests__/) runs first, by its path, once siftwise can tell the path Jest expects from a known Jest test in the same directory, a parent or a matching project-relative directory. Otherwise, and for a new file outside those patterns, the full suite runs as before. Shadow mode can't credit the tests in a new Jest file (it doesn't know their IDs yet): a failure there counts as a new_test miss.

Results on real repos

Replays of real CI history with no peeking at the future, with the current code except where noted. How they measure, and what they don't prove: eval methodology; full numbers: eval results.

Repo

Failures caught

Test time skipped

ipfs/kubo (Go, ~4,100 tests), measured with the week-2 code

19 of 19 in failing PRs

30.7% over 20 main commits replayed as PRs; 0% on recent PRs (all dependency/CI bumps)

RDFLib/rdflib (Python, ~18,000 tests)

2,049 of 2,050 in 39 failing PRs; 140 of 141 in selective runs. The miss is a flaky remote-service test

10.4% on the 10 most recent PRs; 0.6% over 19 main commits

quantumlib/Cirq (Python monorepo, 9 packages, ~40,000 tests)

not measurable: JUnit only from main, which never failed

37.9% over 132 merged PRs; 79.8% for cirq-google-only PRs, 20.2% for cirq-core

agent-substrate/substrate (Go, ~9,000 tests, 8 days of history)

2,043 of 2,044 in 72 failing PRs; 38 of 39 in selective runs, with declared rules for its e2e and functional suites (26 of 38 without them, week-3 code)

5.1% on PR runs and 13.9% over 40 main commits, with both rules (19.5% / 31.7% without rules, week-3 code)

  • Savings follow package boundaries. In the Cirq monorepo, a change to one integration package skips most of the suite; a change to the core package everything imports skips little. rdflib's __init__.py and plugin registry connect nearly every module to nearly every test, so safe file-level selection still runs ~99% of its suite (its 10.4% comes from lint-only PRs). Bigger savings need codebases with independent parts, or per-test coverage.

  • The misses: substrate's are end-to-end suites that drive a deployed system (no import edge reaches them; a declared dependency catches them at the cost of most of the savings) and flaky tests. rdflib's one miss, test_service_with_bound_solutions, calls a remote SPARQL service and had failed in unrelated runs before: it's flaky, which is what rerun-unknown is for.

Triage (a verdict per failure, scored against what happened next; details):

  • The worst error, a regression called flaky: 0 on rdflib. On substrate, 5 of 406 labelled regressions, all on tests with earlier same-commit flakes; 4 are likely label noise from merge diffs.

  • The prediction tests held up: new_regression 319 of 319 labelled substrate failures (and 10 of 11 rdflib fix episodes with a known diff), and medium-confidence pre_existing 759 of 759, each called after a single failing main run.

  • Flaky precision is mostly by construction: almost every flaky verdict is a pass of the same commit seen before the failure, which is also the flaky label's evidence.

Performance (rdflib history; results identical before and after; details): triage of a 22-variant CI run 1,032 → 220 ms (180 → 12 queries, no longer growing with variants); the selection inside /select 605 → 237 ms; ingesting a 9,891-result upload about 2.2 → 1.25 s.

How it compares

From each project's public docs as of October 2026; corrections welcome.

Open source

Self-hosted

Languages

How it selects

Agent/MCP support

siftwise

Yes (Apache-2.0)

Yes

Python, Go, JS/TS

file mapping, Go/Python import graphs, failure history

MCP server (select, triage, explain)

CloudBees Smart Tests

No

Not stated

Many (runner integrations incl. Python, Go, Java, JS)

LLM analysis of changes + test history

Not documented

Datadog Test Impact Analysis

No

No (Datadog SaaS)

JS/TS, Java, Python, Go, Swift and more

per-test code coverage

Not documented

Develocity Predictive Test Selection

No

Yes (Develocity server)

Gradle and Maven builds

predictive model from past builds

Not documented

pytest-testmon

Yes (MIT)

Local plugin, no server

Python (pytest)

per-test code coverage

No

Coverage-based tools see dynamic dependencies that static graphs miss, but they need instrumented runs. siftwise uses no instrumentation; when its graphs can't see a dependency, the fallbacks and .siftwise.toml cover it.

Security and limits

Found a vulnerability? Report it privately, as SECURITY.md describes, not in a public issue.

  • One token per server, with full access. SIFTWISE_API_TOKEN is a single shared secret: every client (CI, the CLI, the MCP server) uses it, and it can read and write the history of every repo on the server. There are no per-repo, read-only or per-user tokens. Run a separate server for repos whose owners shouldn't see each other's history, and rotate the token by changing it on the server and in each secret.

  • Keep the token out of fork PRs. Use pull_request, never pull_request_target: a fork's code must never run with the token. On GitLab, never run untrusted forks' merge request pipelines in the parent project (Forks).

  • Protect your workflows. The base-branch known_flaky rule assumes the workflow itself is protected. A PR that edits .github/workflows/ (or .gitlab-ci.yml) can change any gate, so protect workflows with CODEOWNERS or a ruleset (on GitLab: protected branches, and code owner approval with Premium or Ultimate). GitLab's token choices: protected or unprotected variables.

  • What the server stores: test IDs, outcomes, durations, failure messages (truncated to 4 KB), changed file paths, branch names and commit SHAs. Failure messages can contain whatever your tests print, so treat the database as you treat CI logs. Public PR comments show test IDs, verdicts and siftwise's own reasons, never raw failure messages.

  • Stats cover the window only. Per-test stats use the last SIFTWISE_STATS_WINDOW_DAYS (90) days before the repo's newest run; older raw results stay, but no longer feed stats.

  • Limits to know: merge-queue (merge_group), scheduled and manual runs always run the full suite (no diff base); the Action is tested on Linux runners only, and the GitLab template with mocks and one example project on gitlab.com (Free).

Roadmap

  • Done (v0.1): recording and history, test selection with import graphs and safe fallbacks, the GitHub Action, failure triage with verdicts in the PR comment and fail-on, the MCP server, the triage eval, shadow mode with siftwise report, miss causes, flaky handling (known_flaky, rerun-unknown), performance (batched triage, faster select and ingest), packaging (a light CLI, a server image, the quickstart) and the v0.1.0 release on PyPI and GHCR.

  • Done (v0.2): MCP support for more clients (tested with Claude Code, GitHub Copilot in VS Code, Cursor, Codex CLI and Gemini CLI), rerun-unknown for Go, Jest, Maven and Gradle, and GitLab CI support (changelog).

  • Done (v0.3): risk order, time budgets, Jest and Java (Maven and Gradle) selection, import depth in the risk order, module build-file scoping for Java, and live dogfooding numbers: every siftwise PR and push to main runs as a shadow run, with a server inside the CI job (dogfood.yml).

  • Next (v0.4): scoping test-fixture build files (today a fixture's build.gradle.kts or tsconfig.json runs the full suite), and more recall numbers; a demo and the launch write-up; an arm64 server image.

Contributing

Issues and pull requests are welcome: see CONTRIBUTING.md for the dev setup, the checks and the PR checklist, and ARCHITECTURE.md for how the pieces fit. Everyone taking part follows the Code of Conduct.

License

Apache-2.0

Available Tools

3 tools
explainA

Call this to understand one test: its recent results, whether it is flaky or broken on main, and why siftwise would or wouldn't select it for your current change. Use it when a selection or a triage verdict surprises you, before overriding it.

Inputs: test_id (required), base, variant, repo_path, +1 more. Returns: object

select_testsA

Call this before running tests after changing code. It returns the command that runs only the tests your change (committed and uncommitted) can affect, with a reason per test. Run that command instead of the full test suite. If mode is "full", run the project's full test suite; if the command is empty in selective mode, no tests are affected.

Inputs: base, max_tests, repo_path, include_uncommitted. Returns: object

triage_failureA

Call this when tests fail, before trying to fix them. For each failing test it says whether your change caused it (new_regression: fix these), it was already failing on main (pre_existing: not yours), it is flaky (flaky: re-run it, don't fix it), or siftwise can't tell (unknown). Pass the JUnit XML report paths from the test run, or the failing test IDs (pytest node IDs like tests/test_x.py::test_y work as classname::name, e.g. tests.test_x::test_y) with optional failure messages.

Inputs: base, variant, failures, repo_path, +2 more. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4/5.0

    Scored across 3 tools

    Disambiguation5/5

    The three tools serve clearly distinct workflow stages: selecting affected tests before a run, triaging failing tests afterward, and explaining one test in detail when needed. The descriptions explicitly state when to call each, so an agent should not confuse them.

    Naming Consistency4/5

    select_tests and triage_failure follow a clear verb_noun pattern, while explain is a bare verb. The deviation is minor and readable, but it prevents a perfect consistency score.

    Tool Count4/5

    Three tools is a minimal but well-scoped set for the server's test-selection and triage purpose. Each tool earns its place, though the surface is at the low end of what feels comfortably complete.

    Completeness4/5

    The tools cover the core lifecycle: selecting tests, triaging failures, and explaining individual test behavior. Minor gaps exist around discovering/configuring test inventory or reporting outcomes, but the main workflow has no dead ends.

    Maintenance

    ActivityActive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers