A suspected bug in simulation code is a claim. physverify turns it into a run against a reference written from the physics, and gives a verdict with the numbers.
The claim can come from an AI code review, a colleague, a static check or your own doubt. Reading the code again rarely settles it; a run against an exact solution, a conservation law or a convergence order does. physverify supplies those checks for numerical ODE and PDE code (order of convergence, manufactured solutions, conservation, Runge-Kutta tables, exact references, a few Monte Carlo statistics), a sandbox for running code you do not trust, and plain-language verdicts. It needs only NumPy. Current release: 0.8.0 (v0.8.0); actively developed, contributions welcome.
Five AI reviewers (three Claude models, Qwen 3.8 Flash and GPT 5.6 Terra) reviewed 100 public physics repositories with one fixed prompt. Fifty had a defect reported earlier; fifty had been tested and found clean. On the clean fifty, 119 review claims were drawn and each was given a check run in the sandbox:
40 claims were real defects that change a result: 33 distinct defects in 19 of the 50 "clean" repositories. 32 have been reported to their maintainers.
51 were real but minor (invalid input, display, docstrings, code paths the repository never uses), 10 were wrong and 18 could not be settled.
Nine of the ten wrong claims came from one model. Where a reading was recorded before the run (20 claims), the run changed 3 verdicts.
Against the 67 defects reported earlier, the reviewers named 36 to 49; physverify scan, which reads code without running it, named 11.
So AI review finds many real defects, including ones earlier testing missed, but a claim is not a finding until something runs. The counts hold for this sample only and are not a bug rate for physics code. The labels come from two automated checkers following written rules, not from people. Tables: scripts/study100.py regenerates them from the per-claim file, released with the preprint (doi:10.5281/zenodo.24224433).
Related MCP server: research-check
Some defects confirmed or fixed by the projects' own maintainers after a report:
Project | What was found | Outcome |
three units differ from their exact definitions | confirmed against primary sources | |
"surface tension" registered as J m² instead of J/m² | confirmed; fix in progress | |
strings with a space treated as datetimes | fixed | |
de Sitter scale factor on a spatial coordinate | fixed | |
entropy pressure term with the wrong units | fixed | |
two constants modules defaulted to different unit systems (CGS and SI), so | fixed | |
Runge-Kutta schemes labelled one order too high | fixed; now uses |
The full list has more projects and the details.
pip install physverify
physverify demoThe demo checks a correct solver (all checks pass), then the same solver with the damping 10% too large (all fail):
New to this? Start here takes about ten minutes and includes a prompt to paste if you use an AI coding assistant.
Suspected bugs from a review (yours or an AI's): write each claim down, pick a reference that does not come from the code, and run the smallest test that tells the two answers apart. Checking an AI code review walks through it with examples.
An ODE solver: physverify init verify_solver.py --solver mysolver --order 4 writes a spec; fill in the equations (from the physics, not from your code) and run physverify run verify_solver.py.
Errors you already have: check_order(hs, errors, expected=2) gives PASS, FAIL or INCONCLUSIVE with a reason.
A Runge-Kutta table: check_tableau(A, b, c, claimed=3) from physverify.butcher checks order, stage order and stability from the coefficients alone.
Code you have not read yet: physverify scan PATH reads the source, runs nothing, and flags a fixed set of patterns (Runge-Kutta tables below their stated order, grid spacings that do not match their grid, near-miss constants, removed NumPy/SciPy features, undefined names, unit tables with the wrong dimension). A clean scan means only that those patterns were not found. scan is kept working but no longer gets new rules; the work now goes into confirming claims.
Code from other people runs only in the sandbox (no network, read-only repository).
A failed check fails the job, like a broken unit test (CI guide):
- uses: rcgiri-physics/physverify@v0.8.0
with:
specs: tests/verify_*.pyGetting started · Recipes · What a PASS means · Troubleshooting · API
physverify mcp is a local MCP server with two tools: run_checks runs a project's verify_*.py specs, and confirm_claim judges one suspected defect (confirm). Install physverify, then add this to your client's MCP settings (Claude Desktop, Cursor, Windsurf, Cline, Qoder and others; VS Code, Codex CLI and Claude Code are in the guide):
{
"mcpServers": {
"physverify": { "command": "physverify", "args": ["mcp"] }
}
}The tools run your project's Python code on your computer, so use them on projects you trust. The server is in the official MCP registry as io.github.rcgiri-physics/physverify.
Seeded bugs: of 37 bugs added one at a time to five small solvers, the checks flagged 35 (plain "compare with one answer" tests: 27), with no false alarms. The same person wrote the bugs and the checks, so this shows the tool works as designed, not how it does on other code. python scripts/reproduce_all.py regenerates the tables.
Earlier corpus testing (October 2026): of the first 74 defects reported to authors, 14 were found by a physverify check as such, 11 by running the code at all, and the rest by the reference in each repository's test script or by reading; the ledger lists them by ID. Details: results.
Works best for ODEs; PDEs only in 1D so far. Your code needs a way to add a source term, or a small wrapper that can itself be wrong. A PASS means these checks found nothing wrong; it does not prove the simulation correct, and it is only as independent as the reference you wrote: see What a PASS means.
Help is welcome: a trial report on your own solver, seeded bugs written by someone other than the author, adapters, documentation. See CONTRIBUTING.md, the code of conduct, SUPPORT.md, SECURITY.md on running untrusted code, and AGENTS.md for AI assistants. To cite, use doi:10.5281/zenodo.23534512 or CITATION.cff. Changes: CHANGELOG.md; plans: ROADMAP.md. The methods are standard and other tools do parts of this, some better (NodePy for Runge-Kutta analysis); see related work and credits. MIT licence, copyright (c) 2026 Ram Chandra Giri.
Judge one suspected defect with physverify confirm. Takes a claim file that defines CLAIM, control() and checks() with a reference written from the physics (docs/confirm.md), runs it in a fresh process and returns the verdict: confirmed, rejected or not decidable, with the reason and each check's status and message. Appends the result to a ledger file. Runs the code, so it needs trusted=true or the sandbox (PHYSVERIFY_REPO set).
Inputs: claim (required), ledger, timeout, trusted, +1 more.
Run physverify verification checks (manufactured solutions, order of convergence, invariants, limits, exactness) on physics simulation code. Takes spec files that define checks(); with no specs given, finds verify_*.py files under project. Returns a JSON report: per check family, label, status (PASS, EXACT, FAIL, INCONCLUSIVE, ERROR, SKIP), message, observed and expected order, step sizes and errors.
Inputs: specs, project, timeout.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 2 tools
run_checks is a broad verification sweep over discovered verify_*.py specs, while confirm_claim adjudicates a single suspected defect from a CLAIM file and writes to a ledger, so purposes are mostly separable. They share the underlying checks() machinery and both report check statuses, which could cause some hesitation about which to call for a one-off verification.
Both tools follow the same verb_noun snake_case pattern (run_checks, confirm_claim) and the verbs accurately reflect their actions. No mixing of conventions.
Only two tools for what is a fairly rich verification domain, which feels thin even for a focused CLI wrapper. The pair covers the batch and single-claim workflows but leaves little room for auxiliary operations.
Core lifecycle of run-specs-then-confirm-defect is present, but the surface is missing useful operations such as listing/discovering available spec or claim files, reading the accumulated ledger, or validating spec syntax before execution. The write-only ledger is a dead end for an agent.
Physics-based validation of simulation results: receipts with per-check verdicts, via MCP.
Proves AI-generated Python does what you asked: lint, types, security, sandbox run, exact fixes.
Verification for AI coding agents: runs the tests and checks a change affects, with a receipt.
Run plain-English web QA tests and inspect check results and evidence.