failgate

🤔 Why

Coding agents now open pull requests at scale — about 17 million a month on GitHub by March 2026. Writing the fix is no longer the hard part. Knowing whether it is right is.

• 🧪 Tests written after the fix grade themselves. When an agent sees a failing test, editing the test is the cheapest way to make it pass — ImpossibleBench caught GPT-5 doing exactly that in 76% of impossible tasks.

• 🕳️ "Tests pass" is a weak signal. UTBoost found that 15.7% of the patches counted as resolved on SWE-bench Verified are actually wrong — the tests were too weak to notice.

• 🔁 Reproduction bots stop too early. Several tools now turn an issue into a failing test. Very few keep that test honest when a PR arrives: was it edited? skipped? does it still fail the same way? did something else break?

FailGate is the acceptance layer: it writes the exam before the fix exists, locks it, and grades every claimed fix — from a maintainer, a contributor, Claude Code, Codex, Copilot, or its own fixer agent — the same way.

Related MCP server: Claude Code Bridge

🧭 How it works

Wording matters: a pass means "passes the acceptance test + no tampering found + no new regressions", not "the fix is correct".

📊 Results

Datasets are small and every number has known limits (see What FailGate does not claim); the human-style reviews were done by Claude, not by the projects' maintainers.

⚡ Try it in a minute

Needs Python 3.11+, Docker, and a GitHub token (read-only is enough). No LLM key needed — the demo exams are already sealed.

git clone https://github.com/san086041-glitch/FailGate && cd FailGate
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\Activate.ps1
pip install -e .
export GITHUB_TOKEN=$(gh auth token)

python scripts/media/demo_exams.py import             # the demo repo's public sealed exams → your local DB
failgate verify san086041-glitch/failgate-demo#5      # a PR that only edits the test → refuted
failgate verify san086041-glitch/failgate-demo#16     # a real fix → accepted, with exam strength

Then type failgate for the home screen and interactive shell, or failgate doctor to check your setup.

Just the CLI, without the demo data, is on PyPI:

pip install failgate
failgate --version
failgate doctor          # what is configured and what is missing

🧩 Three ways to use it

🐙 GitHub App

Install on a repo. New issues get triage, dedup, and — for bugs — a sealed failing test. PRs that say Fixes #N get a verification report.

Maintainer commands: /failgate verify, /failgate reseal, /failgate fix.

New repos start in shadow mode: every write is logged, nothing is posted.

Setup →

⌨️ CLI

failgate verify owner/repo#123
failgate checkup owner/repo
failgate up

checkup runs the whole evaluation on any public repo and writes a plain-language report. up starts the service + webhook tunnel with a live status board.

failgate --help lists every command.

🤖 MCP (local, stdio)

Let Claude Code or Cursor use FailGate as their acceptance gate:

claude mcp add failgate -- \
  python -m failgate mcp

Tools: reproduce_issue, run_acceptance_test, verify_fix, get_fix_task. Nothing is written into your repo.

Long-running deployment: docker compose up -d --build brings up the API, a sandbox worker, PostgreSQL and Redis — see docs/deploy.md.

🔁 End to end: from issue to merged fix

One real run on the demo repo (issue #21 → PR #22). The only human actions were opening the issue, commenting /failgate fix, and clicking merge.

About 10 minutes end to end, $0.013 in LLM calls. Here is the same run from the operator's side — failgate up keeps the service and webhook tunnel running and shows every state change:

🔧 Under the hood

• Explicit state machine, not a free-running agent. Each issue / PR is a Case; agents run inside bounded steps with budgets.

• One door for side effects. Every GitHub write goes through PolicyGate: idempotency keys, shadow mode, secret scanning, label policy.

• Separated roles. The exam writer, the fixer and the grader have different permissions: the fixer cannot touch tests or test config (enforced in the tool layer), and pushes through a separate GitHub App.

• Sandbox. Two phases: install through an egress allow-list proxy, run with no network, non-root, all capabilities dropped, read-only root, pid / memory / CPU limits.

• Platform. Two queue lanes so sandbox jobs never block triage (fast-event p95 448 s → 22 s), Redis + arq with crash redelivery, PostgreSQL + Alembic, OpenTelemetry traces to Langfuse.

• Measure first. Every component has an offline replay on real repo history ("time travel" to the commit before the fix), and every design decision is written down (what was tried, what failed and why) before the code changes.

Stack: Python 3.11 · FastAPI · SQLAlchemy 2 (async) · PostgreSQL / SQLite · Redis + arq · Docker SDK · LangGraph (fixer agent) · cosmic-ray (mutation) · bm25s + bge-m3 (retrieval) · OpenTelemetry · MCP Python SDK · Typer + Rich + prompt_toolkit.

🚫 What FailGate does not claim

• "Verified" ≠ correct. A weak exam lets a wrong fix through — that's why strength and hidden exams exist, and why they are shown, not hidden.

• Small, honest samples. 4 Python repos, 12 sampled issues each, one run each; human-style review was done by Claude with a "GitHub evidence required" rule.

• Python + pytest only, for now. Packages that need system libraries may not install in the sandbox. Monorepos work when the package under test lives in one subdirectory (--subdir libs/core, e.g. LangChain). More languages are on the roadmap.

• Exam strength is a hint. It reacts to stronger tests (10 up / 0 down on SWE-bench + UTBoost, p = 0.002) but can't by itself tell a weak exam from a good one (p = 0.41).

🗺️ Roadmap

• More languages. FailGate works on Python + pytest today. The exam → seal → grade design does not depend on the language; what is Python-specific is four adapters — sandbox install, test runner, failure signatures and mutation testing. Next up: JavaScript / TypeScript (Jest, Vitest), then Go and Java.

• More platforms. Gitee, alongside GitHub.

• A v0.1 release with a recorded walkthrough.

📚 Docs

• Deploy with docker compose

• Set up the GitHub App

• Set up the Fixer App (only needed for /failgate fix)