bite

Would your tests catch it if the bug came back?

Bite checks whether an AI coding agent's "fixed" is actually true.

Try it now

Free, no key, and it never changes your files. If you have Python, paste these two lines into any terminal (PowerShell, cmd, Terminal):

python -m pip install bite-check
python -m bite_check --free

Outside a project, Bite first shows a demo on a tiny made-up one. To check your own project, type the second line and a space, drag your project's folder into the window (that types its path), and press Enter.

Other ways: pipx run bite-check --free or uvx bite-check --free (one line, no install; on Linux or a Mac, use one of these if pip says "externally managed"). Commits never say "fix"? Add --any. To keep a bite command installed: install guide.

Related MCP server: platao

Why you need this

When an AI fixes a bug for you, it often says "Fixed, and I added a test." If you don't read code, you can't tell if that's true.

A test is a tiny robot that checks one thing and shouts when it's wrong. Bite puts the old bug back on purpose and watches whether any test shouts. It's like testing a smoke alarm with real smoke instead of trusting the little green light.

If a test shouts, that fix is guarded. If they all stay quiet, the bug could come back and nobody would notice. Bite names that fix, so you can tell your AI: "add a test that fails without this fix."

bite --demo shows both cases on a tiny made-up shop:

  9661edcf ✅ noticed by test_discount_never_below_zero Fix discounts over 100% making the price negative
  e5c329ec ⚠️ NO TEST NOTICED           Fix totals showing long decimals like 0.30000000000000004

Bite does this: takes each recent fix back out of a copy of your project, runs your tests, and tells you whether any test noticed. It never changes your files or pushes anything.

You decide that: whether to add the missing test, or leave it because the change didn't need one (a rename, a tidy-up). A ✅ means a test would catch that fix breaking. It doesn't prove the code has no other bugs.

Works with pytest, npm test, cargo test, go test, dotnet test, Maven and Gradle (Java and Kotlin). Full list: SUPPORTS.md.

Everything else Bite can do

The pull request check. Add this as .github/workflows/bite.yml. It runs the free check on each pull request's commits and fails when a change can be taken back out and your tests still pass. No key, no cost.

name: bite
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  bite:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0
      # install what your tests need here, e.g. for Python:
      - uses: actions/setup-python@v7
      - run: python -m pip install pytest -r requirements.txt
      - uses: connabyss/bite@v1.0.5

The paid step (optional). With your own Anthropic API key, Bite writes one small test of a commit's exact claim and runs it; --budget caps what a run spends ($1 by default). It runs on Anthropic's Agent SDK, so it needs an API key and never uses a Claude login or plan (Anthropic's Agent SDK rule: https://code.claude.com/docs/en/agent-sdk/overview). Setup: install guide.

Updating Bite

bite --version says when a newer one is out. This line always gets the newest version:

pipx install --force bite-check --pip-args="--no-cache-dir"

Installed with pip? python -m pip install --upgrade --no-cache-dir bite-check. Why not pipx upgrade: install guide. What changed: CHANGELOG.md.

Does it work?

In prosemirror-py, a Claude commit said it fixed a memory leak. Bite wrote a test of that claim: it fails on that commit and on today's code. After 20,000 documents the cache still held all 20,000, using 35 MB. The tested fix was merged upstream. (The full story.)

"Reverting the claimed fix and making the change prove itself under a real test is exactly the kind of check that should exist for every "fixed" commit."

— Ivan Zdravkov, on DEV

Famous repos Bite has checked with bite --free (Oct 30-31, 2026):

More: the full scoreboard (20 projects, 5 languages), and bugs fixed or merged upstream in pyclibrary, django-tailwind-cli, west, mizani, pamqp and more.

Limits, plainly

• Most claims hold. Across all rounds so far, Bite tested 137 "fixed" claims that it could run tests for (all from commits made with Claude Code, the AI those rounds looked at). 8 were false (about 1 in 17), and 4 of those were still broken on today's code. On well-reviewed projects it usually finds nothing, and that's the right answer.

• Quiet is not a pass. Bite only shows a failing test that the second check calls fair and real. That second check has thrown out more than 10 failing tests that weren't. A claim that holds proves that one claim, not that the commit has no other bugs.

• It needs the project's tests to run. If they don't run on a clean copy, Bite can't check that commit.

• Bug hunting is weaker than claim checking. In a 10-project overnight hunt, two blind AI judges rated 21 of 22 proven fixes as real bugs and none as wrong, but Bite found only 1 of the 10 bugs the maintainers fixed soon after.

• Small samples, AI judges. The judges read code without running it. The human checks are the maintainers' fixes and merges listed above.

Join in

Here for Hack Week or your first open-source PR? These good first issues are small, and each one says where to look and when it's done. Comment on one to claim it. Or try to fool Bite: sneak a "fix" past it.

🏆 Fooled Bite: nobody yet. Be the first: your name, your trick, and the test that now catches it go right here. (How to enter. Bite's own 2 known blind spots are in Still fools Bite, so they don't count.)

Bug reports, fixes and AI-assisted PRs are all welcome. See CONTRIBUTING.md. If Bite flagged something wrongly, or missed a claim you know was false, please open an issue.

Thank you to Bite's first contributors:

• @snehaapratap made the setup error messages point to steps that exist (#18).

• @gauravjangra3165 added macOS to the tests (#19).

• @bfs2021 made the "check it yourself" steps work for commits that add a new file (#37, fixing #27).

MIT licensed.