nightward

A regression gate for AI-made changes. Your AI agent can run the gate; opening it takes a human-approved baseline diff.

When an AI agent fixes A, it often moves B, C and D too, and the test suite stays green because no test pins down B, C or D. You find the side effects later, fix them, and the fixes move something else. An agent loop has no reliable way to know when it is done.

nightward gives that loop a definition of "done" that the agent can check but not change:

1. Your tests capture what the code does today with a behavior(...) pytest fixture. You don't write expected values.

2. A human reviews those captures and approves them once. That baseline is a set of JSON files committed next to your code.

3. After every change, nightward captures again and lists everything that moved, grouped by feature (the blast radius). The gate stays closed until a human approves each change as intended or the code is fixed.

The agent gets run and status, on the command line or as MCP tools. Approving is a human CLI command, and its result is a file diff that goes through code review.

nightward run       # capture behaviors and compare them with the approved baseline
nightward review    # a human reads the diffs
nightward approve   # the human accepts what review showed (intended changes only)
nightward gate      # exit 0 only when nothing unapproved remains (CI, agent loops)

nightward is not a test generator, and it has no idea what is correct. It only makes sure that no change to a captured behavior passes silently.

What's new in 0.3, and what changes for 0.2 users: see the CHANGELOG.

How a project uses it

Setup happens once. After that, every change goes around the loop, and only a human can take an "approve" edge:

flowchart TD
    subgraph setup["Once per project"]
        A["Write capture tests<br/>behavior(name, value, group=)"] --> B["nightward run"]
        B --> C{"A human reads the first capture<br/>nightward review"}
        C -- "a value is wrong" --> D["Fix the code"] --> B
        C -- "this is what it should do" --> E["nightward approve<br/>commit .nightward/baseline/"]
    end
    E --> F
    subgraph loop["Every change, by an agent or a human"]
        F["Edit the code"] --> G["nightward run<br/>(MCP: nightward_run)"]
        G --> H{"boundary"}
        H -- "intact" --> I["Done: open the PR"]
        H -- "breached, not intended" --> F
        H -- "breached, intended" --> J["The agent stops and lists the changes"]
        J --> K{"A human runs nightward review"}
        K -- "approve" --> L["Baseline diff in the PR,<br/>reviewed by code owners"]
        K -- "reject: a regression" --> F
    end
    I --> M["CI: nightward gate<br/>exit 0 only when intact"]
    L --> M

The same loop as a timeline, for an agent change that moves something it shouldn't:

sequenceDiagram
    actor Human
    participant Agent
    participant NW as nightward
    participant CI
    Human->>Agent: "add a coupon to checkout"
    Agent->>Agent: edits the code
    Agent->>NW: nightward_run
    NW-->>Agent: breached: checkout_total (intended), loyalty_points (not intended)
    Agent->>Agent: fixes the loyalty side effect
    Agent->>NW: nightward_run
    NW-->>Agent: breached: checkout_total only
    Agent-->>Human: "checkout_total changed as asked - please review"
    Human->>NW: nightward review, then nightward approve
    Human->>CI: push (the baseline diff is in the PR)
    CI->>NW: nightward gate
    NW-->>CI: exit 0 (intact)

Related MCP server: agent-invariants

Install

nightward needs Python 3.10+. Installing it also registers its pytest plugin, so the behavior fixture is available in every test.

pip install nightward
pip install "nightward[mcp]"      # + MCP server for AI agents

Other extras: [judge] (the Anthropic SDK, for the LLM judge) and [docs] (the PDF, Word and Excel adapters).

Quickstart (5 minutes)

In an empty directory, create a tiny app and a test that captures its behavior. app.py:

def checkout_total(items):
    subtotal = sum(i["price"] * i["qty"] for i in items)
    tax = round(subtotal * 0.1, 2)
    return {"subtotal": subtotal, "tax": tax, "total": round(subtotal + tax, 2)}


def user_login(username):
    return {"user": username, "status": "ok", "roles": ["member"]}

test_app.py writes no expected values. It captures what the code returns (for anything bigger than this, shape the payload so a human can review it: payload design):

from app import checkout_total, user_login

CART = [{"price": 10.0, "qty": 2}, {"price": 5.5, "qty": 1}]


def test_checkout(behavior):
    behavior("checkout_total", checkout_total(CART), group="billing")


def test_login(behavior):
    behavior("user_login", user_login("alice"), group="auth")

A test that captures one value can return it instead. @capture names the behavior after the test (or name=), and each parametrized case keeps its id:

from nightward import capture


@capture(name="checkout_total", group="billing")
def test_checkout():
    return checkout_total(CART)

1. Approve the current behavior as the baseline.

git init                    # skip in an existing repo
nightward init              # creates .nightward/ and git-ignores its per-run files
nightward run .             # runs pytest and captures; lists what moved (names only)
nightward review            # shows each diff; approve covers only what review showed
nightward approve --all
git add .gitignore .nightward   # commit baseline/, rejected/, judge/ (init ignores per-run files)
git add app.py test_app.py && git commit -m "nightward baseline"

run finds two NEW behaviors, review prints their JSON, and approve makes them the baseline:

Boundary: breached (2 unapproved)
unchanged=0 changed=0 new=2 removed=0
...
approved checkout_total (c9ff6ed9)
approved user_login (b86be5a5)

Boundary: intact
unchanged=2 changed=0 new=0 removed=0

2. Make the kind of change an agent makes. In app.py, change the tax rate 0.1 to 0.2 (in bash: sed -i 's/subtotal \* 0.1/subtotal * 0.2/' app.py). Then:

nightward run .             # the tests still pass
nightward gate              # exit code 1: not done
nightward status --json     # the same verdict, for a script or an agent
nightward approve --all     # refused: nobody has reviewed this change yet
nightward review            # the diff a human decides on

Boundary: breached (1 unapproved)
unchanged=1 changed=1 new=0 removed=0

group: billing
  - [CHANGED] checkout_total

boundary breached (1 unapproved) (as of the last run, ...)

{"boundary": "breached", "unapproved": 1, "changes": [{"name": "checkout_total", "kind": "CHANGED", "group": "billing"}], ...}

error: checkout_total changed since your review (the capture now in the store was taken by `nightward run` at ...). Run `nightward review checkout_total`, then approve what it shows.

[CHANGED] checkout_total
--- approved
+++ received
@@ -1,5 +1,5 @@
 {
   "subtotal": 25.5,
-  "tax": 2.55,
-  "total": 28.05
+  "tax": 5.1,
+  "total": 30.6
 }

3. Decide. If the change is a regression, fix the code: put 0.1 back, run nightward run . again, and it reports Boundary: intact, so nightward gate exits 0. If the change is intended, run nightward approve --all (it now succeeds, because review showed you this diff) and commit the updated .nightward/baseline/checkout_total.approved.json.

4. Optional: see the blast radius in a browser. nightward view builds a static, read-only dashboard and serves it on localhost. nightward view --no-serve only builds nightward-site/, which init already git-ignored.

For a bigger cascade (one tax tweak moves three customer-facing behaviors), run the petshop example from a clone of this repo.

What to commit

In CI: the GitHub Action

Set up Python and your project's dependencies, then gate the PR with one step. It runs your tests, writes the verdict and the diffs that moved to the job summary, and fails the job unless the boundary is intact. It never approves: intended changes are approved locally (nightward review, nightward approve) and arrive in the PR as a baseline diff.

- uses: actions/checkout@v4
- uses: actions/setup-python@v5
  with:
    python-version: "3.12"
- run: pip install -e .          # your project and its test dependencies
- uses: LunenLabs/nightward@v0.3.4
  with:
    path: tests                  # what pytest runs; default: [tool.nightward] path, else "."

Inputs: path, dir (default .nightward), pytest-args (one per line), allow-not-run, version, install, summary; output: boundary. A full workflow with the dashboard artifact is in the CI recipe.

Commands

gate and status report the last run, so run nightward run again after every code edit (or use nightward gate --run). The full reference (removals, stale reports, merge conflicts, locking, and using the plugin inside an existing pytest job) is in docs/commands.md.

AI agents (nightward mcp)

An agent loop needs a definition of "done" that it can check but not change. nightward mcp is a stdio MCP server with two tools: nightward_run (run the tests, capture, recompute the verdict) and nightward_status (read the last verdict). approve and reject are not exposed: if the agent that makes a change could also approve it, the gate would turn into a changelog. The judge and the store are the humans' committed choices, so the agent can't pick a lenient judge or point the server at another store.

In Claude Code, install the plugin. It adds the MCP server and a skill that tells Claude to run the gate after its edits and to hand intended changes to you instead of approving them. A second skill, onboard, puts new code under the gate: Claude reads the code without changing it, writes a domain map and the capture tests, and hands you the suspected defects and the first review. nightward itself must be installed in the environment you start claude from (your project's virtualenv):

pip install "nightward[mcp]"                      # in the project's environment

/plugin install nightward --marketplace LunenLabs/nightward

/nightward:status then shows the verdict and what you need to review. Any other MCP host can run the server directly:

claude mcp add nightward -- nightward mcp         # or your host's equivalent; run it in the project root

Both tools return the status --json shape. The agent is done only when boundary is intact and stale is false:

Setup for other MCP hosts, every argument and returned field, and the rules for the loop are in docs/mcp.md.

Gating AI text: the semantic judge

Free-text AI output changes its wording on every run, so an exact comparison breaches every time. Mark such a behavior semantic=True and commit a judge in pyproject.toml: [tool.nightward] judge = "anthropic:claude-haiku-5-5". The key-free persona:* judges are deterministic stand-ins for tests and CI: they forgive changes of case, whitespace and punctuation in prose, not a reworded sentence, so reworded AI text needs the LLM judge. The judge only decides whether two outputs mean the same thing; approval stays human. Every ruling is recorded in the committed ledger .nightward/judge/, and a judge that can't rule falls back to the exact comparison. Details are in docs/semantic-judge.md.

How it compares with snapshot testing

nightward is snapshot testing underneath: capture, approve, compare. If you already use syrupy, inline-snapshot or ApprovalTests and people make your changes, they may be all you need. nightward is built for the loop where an agent makes the change:

Where they're better: inline-snapshot keeps the value next to the assertion, syrupy has serializers for many types and needs no extra command, and ApprovalTests exists for many languages. They can run side by side with nightward in one suite.

What "intact" means

"Boundary intact" means no captured behavior changed, nothing more:

• The gate sees only what behavior() captures, so capture the behaviors you care about.

• The gate is a convention, not a sandbox. An agent with a shell could run approve or edit the baseline, but every bypass leaves a trace in git. Make that trace need a human: own the store with CODEOWNERS, require code-owner review on main, and let CI run the gate from source (recipe).

• A semantic judge can be wrong, so its rulings are recorded for review.

• Normalization defines "the same": timestamps and UUIDs are masked by default (opt out with scrub=False), and dict key order is ignored.

Read the threat model and what nightward normalizes before you rely on a green gate.

Documentation

• CHANGELOG: what's new in 0.3.0, and upgrading from 0.2

• Commands and workflow: every command, removals, stale reports, merge conflicts, and the plugin in an existing pytest job

• Designing payloads a human can review: aggregate by cause, record each input once, keep the machine out - read it before your first capture

• What nightward normalizes: scrubbing, custom scrub rules, names, non-JSON values, and doctor

• Review depth: per group, how much a review shows and when the agent asks - never whether a human approves

• Domain maps: the rules each behavior guards, shown next to its diff; the onboard skill writes them

• Semantic judge

• AI agents (MCP)

• Dashboard

• Document & artifact inputs: PDF, Word, Excel, text and raw bytes

• Threat model

• CI recipe: GitHub Actions

• Recipe: make "only a human approves" hold on GitHub: CODEOWNERS for the store, code-owner review, and an agent identity that isn't an owner

Scope

In scope: pytest capture, the blast-radius diff, the gate, the loop signal, scrubbing and doctor, the static dashboard, the MCP agent gate (run and status, never approve), and the semantic judge. Out for now: PR-comment summaries, call-graph grouping, and other languages. See CONTRIBUTING.md.

MIT licensed.