Checkpoint

Prove your agent works before your customers find out it doesn't.

Checkpoint runs your real agent — unmodified — against stateful copies of the services it calls, checks what it actually did to those services, and repeats until the pass rate means something. Then it ships or blocks the build.

pip install git+https://github.com/baliutkarsh2/checkpoint   # PyPI release pending
checkpoint demo

File an issue  github · no API key · no network
  ✓ [D]  Exactly 1 issue was created
  ✓ [D!] No issues were deleted
  ✓ [T]  The agent made at most 10 calls

  100/100  2 API calls · 3.3s

That took three seconds, sent nothing to the internet, and called no model. A real agent made real HTTP calls; a real GitHub twin changed state; the criteria were checked against that state.

Why this exists

Three things go wrong when you test an agent the usual way.

One run tells you almost nothing. Agents are stochastic. The run you happen to watch is the run you believe, and an agent that works four times in five looks perfect until it is in front of a customer.

Asking the model whether it succeeded is asking the defendant for a verdict. A judge reading the final answer scores what the agent said. Agents say they filed the ticket, issued the refund, sent the message. Checkpoint scores what changed in the service.

Mocks test the mock. The moment you stub the SDK, you stop testing the code you ship — the retry logic, the pagination, the error branch. Checkpoint intercepts TLS locally and routes https://api.github.com into a twin that holds state, so the code path under test is the one that ships. No Docker, no changes to your agent.

Recordings have a sharper problem than staleness: a recording is a sequence, and it replays correctly only if the calls come back in the order they were taped. An agent decides what to call next while it runs. Retry once, poll twice, take the other branch, and the tape no longer lines up. A twin has no order to get wrong — it holds state and answers whatever is asked, in whatever order, as many times as you like. That is the property a non-deterministic caller needs.

Related MCP server: Unity MCP Test Loop

Test your agent

cd your-agent-repo
checkpoint init --command "python my_agent.py"
checkpoint run

init writes three files and touches nothing that already exists: checkpoint.toml, a starter scenario, and a .gitignore line. The starter scenario is all assertions, so that run scores 100/100 with no API key — the same as the demo. There is no harness, no wrapper, no adapter: Checkpoint runs the command that already runs your agent, puts the task in $CHECKPOINT_TASK, and reads the final answer from stdout.

Your agent takes the task another way? --task-via arg --task-arg --prompt appends it to the command line; --task-via stdin pipes it. It is an HTTP service? Put url = "http://127.0.0.1:8000/chat" under [agent]. It logs to stdout? Write the answer to $CHECKPOINT_ANSWER_FILE instead.

Your agent does not have to be Python. Checkpoint starts a command and intercepts the network underneath it, and neither of those cares what the process is written in:

checkpoint init --command "node agent.js"       # or: go run ., cargo run,
checkpoint run                                  # bun start, deno task, ./agent

The proxy hands the agent the environment each runtime actually reads — HTTPS_PROXY in both spellings, SSL_CERT_FILE for OpenSSL stacks, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, HTTPLIB2_CA_CERTS, NODE_EXTRA_CA_CERTS, GIT_SSL_CAINFO, DENO_CERT, CARGO_HTTP_CAINFO and GRPC_DEFAULT_SSL_ROOTS_FILE_PATH — so an unmodified client reaches the twins whatever it is written in. Python, Node and curl are covered end to end by the test suite; the rest are configured the way each runtime documents. Only the judge for [P] criteria and the library API are Python; the thing under test is a process.

A scenario

One markdown file. The task, and what has to be true afterwards.

---
twins: [github]
seed: small-project
---
# File a bug

## Task
File an issue in acme/webapp titled "Login broken".

## Criteria
- [D] Exactly 1 issue was created
- [D!] No issues were deleted
- [T] The agent made at most 6 calls
- [P] The final answer quotes the issue number

[D] checks the state the agent left behind, [T] the calls it made, [P] what it said. ! marks a criterion that must pass whatever the rest score.

Only [P] is judged by a model on every run. A [D] or [T] is an assertion, and most are recognised by pattern and cost nothing; one phrased so that no pattern matches is translated into an assertion by a model once, then cached and re-used. Pin it yourself with => and even that first call goes away — checkpoint check tells you which of the three you have before you run anything.

Each criterion becomes an assertion over the run. checkpoint check shows you which one before you spend a single run on it:

[D]   Exactly 1 issue was created      pattern: count(created.github.issues) == 1
[D!]  No issues were deleted           pattern: count(deleted.github.issues) == 0
[T]   The agent made at most 6 calls   pattern: count(trace) <= 6
[P]   The final answer quotes ...      judged: the judge model reads the final answer

Write your own when you want no ambiguity and no model in the loop:

- [D] The issue is still open  =>  count(github.issues[title == "Login broken" && state == "open"]) == 1

The rule that matters: a criterion must fail for an agent that did nothing. "An issue exists" can already be true of the seed. "Exactly one issue was created" cannot. checkpoint check is where a vacuous criterion shows itself.

The verdict

checkpoint run is the loop you stay in while building. checkpoint gate is what CI reads.

 Scenario             Pass   Rate     95% CI      pass^8   Reading
───────────────────────────────────────────────────────────────────
 file-a-bug.md       16/16   100%   [81%, 100%]     100%   stable pass
 refund-flow.md      12/16    75%   [51%, 90%]        4%   flaky

┌─── gate ──────┐
│ CONDITIONAL   │
└───────────────┘

The gate runs every scenario N times and decides from the distribution — a Wilson confidence interval on the pass rate, not one lucky run. pass^8 is the number that tends to land: of the sixteen runs actually observed, the chance that eight of them drawn at random all passed is 4%. That is the question a release asks, and it is far bleaker than the 75% above it.

Two things that are easy to get wrong and this gets right. A perfect run of fewer than 16 runs cannot clear the default bar, so it reports INCONCLUSIVE rather than a green build — 5/5 is not a weak pass, it is not yet evidence. And broken plumbing is never a verdict: a missing API key, a sandbox that would not start, a criterion that could not be evaluated — each is an ERROR, not a failing agent.

Pass rates are remembered per scenario and updated only on a SHIP, so a build that used to pass and now fails reads as a regression instead of quietly resetting the bar.

In CI

- uses: baliutkarsh2/checkpoint@main
  with:
    command: python my_agent.py
    runs: "16"
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Or checkpoint gate directly — the exit code is the whole product. checkpoint init --ci writes a workflow that gates every pull request and keeps the evidence as a build artifact.

Adding this to a project that already exists? checkpoint gate --report-only prints the same verdict and exits 0, so you can see what it says before it can fail a build. It is louder than the || true you would otherwise write, and unlike || true it does not also swallow the ERROR that means Checkpoint itself broke.

What you get to test against

Seven services, running locally, holding state across a multi-step run: GitHub, Slack, Stripe, Linear, Supabase, Discord, Google Workspace. Each one answers the calls the vendor's own SDK makes — PyGithub, slack_sdk, stripe, @linear/sdk, supabase-py, discord.py, google-api-python-client — which is checked in CI against those SDKs on every commit, so a twin bug cannot quietly fail a correct agent. Each exposes a REST surface and an MCP server.

checkpoint twins list shows them and the datasets they ship with.

A scenario can run against several of them at once, and assert across all of them. This is where agents actually fail — the work spans two systems, the first half lands, the second does not, and the agent reports success anyway:

---
twins: slack, stripe
seed: slack=engineering-team, stripe=subscription-heavy
---
- [D]  The refund is for the full 999 cents
  =>   count(created.stripe.refunds[amount == 999]) == 1
- [D!] No other payment was refunded
  =>   all(stripe.payment_intents[id != "pi_sh_006"], amount_refunded == 0)
- [D]  It quotes the refund id in #engineering
  =>   count(created.slack.messages[text ~ /re_[A-Za-z0-9]+/]) == 1

One run, both services, one verdict over the state each was left in. A refund issued and never announced fails. So does an announcement of a refund that was never issued.

They also misbehave on request, which is the part you cannot rehearse against a real API: rate limits, permission denials, read-only mode, latency, a seeded error rate, or a targeted failure on one specific call.

checkpoint run --rate-limit 5         # the API starts refusing after 5 calls
checkpoint run --read-only            # every write is refused, and attempting one fails the run
checkpoint run --egress none          # the agent cannot reach anything but the twins

Testing something we do not ship? Point Checkpoint at an ASGI app of your own and it becomes a twin like any other:

[twins.billing]
app = "mycompany.testing.billing_twin:app"
domains = ["api.billing.internal"]

Agents that edit a repository

Not every agent calls an API. Point a scenario at a fixture directory and your agent runs inside a throwaway copy of it, with the diff it leaves behind as the thing you score:

---
workspace: fixtures/small-repo
---

- [D] Exactly 1 file was created  =>  count(created.workspace.files) == 1
- [D!] poetry.lock was not modified
- [D] src/app.py defines main  =>  count(workspace.files[path == "src/app.py" && content ~ /def main/]) == 1

Your agent needs no changes: its working directory is the tree. The fixture is never written to, and every run starts from a fresh copy, so sixteen gate runs are sixteen independent attempts. A workspace is a disposable tree and a diff, not a jail — see the docs for what that does and does not protect you from.

Beyond the happy path

checkpoint redteam            # adversarial scenarios, mapped to OWASP Agentic categories
checkpoint simulate refund.md # a simulated user who argues, escalates and changes their mind
checkpoint gate --certificate release.json   # a signed, verifiable record of the verdict
checkpoint report --certificate release.json # the assurance document a reviewer asks for

checkpoint redteam reports which class of attack lands — a prompt injection hidden in tool output, a destructive instruction, an exfiltration attempt — and keeps four outcomes apart that are easy to blur into one:

Only the first is a pass. The distinction in the middle two is the reason each scenario pairs its attack with a real task: an agent that answers "I won't do that" and stops has not demonstrated resistance, and saying so keeps "your agent is exploitable" apart from "your agent refuses legitimate work" — two different problems with two different fixes.

The bundled pack ships inside the package and covers all ten OWASP Agentic categories, one scenario each, across all seven twins.

How it works

checkpoint.toml → Agent          the command that already runs your agent
                  Sandbox        twins + a TLS intercept proxy + an egress policy
                  criteria       compiled to assertions over what changed
                  judge          only for [P], only when one is needed
                  verdict        a pass rate with a confidence interval

The intercept proxy is Checkpoint's own: it mints a local CA, serves per-host certificates, and hands your agent the environment every major HTTP client respects. Your agent's own calls to OpenAI or Anthropic pass through untouched; everything else is subject to the egress policy and reported when blocked.

Everything the CLI does is importable:

from checkpoint import Agent, parse_file, run_scenario

result = run_scenario(parse_file("scenarios/refund.md"), Agent(command="python my_agent.py"))
print(result.score, [c.text for c in result.criteria if not c.passed])

There is a pytest plugin too, so a scenario can be an ordinary test:

def test_refund_flow(checkpoint_run):
    result = checkpoint_run("scenarios/refund.md")
    assert result.score == 100, [c.text for c in result.criteria if not c.passed]

Install

pip install git+https://github.com/baliutkarsh2/checkpoint
export OPENAI_API_KEY=sk-...     # only for [P] criteria; assertion-only scenarios need nothing

Python 3.11 or newer. Nothing to build, nothing to run alongside it. Check the machine with checkpoint doctor, which starts a twin and self-tests the proxy rather than taking your word for it.

Not on PyPI yet. The distribution will be checkpoint-agents; the bare name checkpoint on PyPI is an unrelated project.

Any judge model. Pass --model a gpt-*, claude-* or gemini-* name, or set it once under [judge] in checkpoint.toml. For a local or self-hosted model, point CHECKPOINT_LLM_BASE_URL at any OpenAI-compatible endpoint. Claude needs the anthropic extra (pip install "checkpoint-agents[anthropic] @ git+https://github.com/baliutkarsh2/checkpoint" until the PyPI release); the rest need nothing extra.

Where it is honest about itself

• The twins reproduce the endpoints scenarios exercise, not every corner of every API. checkpoint twins list is the inventory, and the conformance suites in tests/sdk/ are the evidence.

• A [P] criterion is a model's opinion. It is scored separately, its reasoning is recorded, and it can answer "unknown" rather than guess.

• checkpoint redteam generate writes attack candidates. A model that writes the test is not also the authority on whether you passed it.

• A run that could not be scored is never counted as a pass or a failure.

Commands

checkpoint init      point Checkpoint at your agent
checkpoint demo      see it work — offline, no API key
checkpoint run       run scenarios against your agent
checkpoint gate      decide whether this build ships
checkpoint redteam   run adversarial scenarios
checkpoint simulate  hold a conversation as a simulated user
checkpoint new       write a new scenario
checkpoint check     check scenarios before you run them
checkpoint twins     the services scenarios run against
checkpoint cert      verify and read signed verdicts
checkpoint report    build an assurance report
checkpoint runs      past runs: list, show, compare, export
checkpoint view      open the dashboard
checkpoint mcp       serve Checkpoint over MCP
checkpoint doctor    check this machine

checkpoint mcp puts all of this inside your coding agent: any MCP client can list scenarios, run one, and gate the build while it writes the very agent under test.

Documentation

Getting started · Scenarios · Twins · The gate · Troubleshooting · Architecture · Self-hosting

Contributing

Issues and pull requests are welcome — see CONTRIBUTING.md. Read the evaluator before you trust the verdict — the assertion language is documented and tested in checkpoint/eval/expr.py, and every criterion's assertion is stored with the run.

Apache-2.0. See LICENSE and SECURITY.md.