Skip to main content
Glama
README.md
# Buggy

Finds a real failing input for a Python function by executing it, verifies the
failure three ways, and hands you a certificate naming the exact call that
breaks it.

```
$ buggy investigate budget_usage -f budget.py --post 'math.isfinite(result)'

  budget_usage  (budget.py)
  ----------------------------------------------------------------------
  status      proven

  Certificate of failure
    input       [0, 0]
    broke       must not raise ZeroDivisionError
    detail      ZeroDivisionError: division by zero
    reproduced  3/3

  Call it yourself to see it fail:
    budget_usage(0, 0)
```

**Pure Python, no dependencies, and no Node.** Everything here is standard
library. This project began as a TypeScript tool; that implementation was
removed once this one surpassed it, and is preserved at the `ts-final` tag.
See [History](#history) for what happened and why.

**Status: early.** Proving, repair, recording and the MCP server work and are
tested. See
[What exists today](#what-exists-today) - that section is the project's
credibility and is kept honest.

## Install

Python 3.10 or newer is the whole requirement.

Install into a virtualenv and link the two commands onto your `PATH`. The
virtualenv keeps the tool from touching the environment of whatever project you
point it at; the symlinks mean nothing holds a venv path, so the venv can be
rebuilt without breaking your editor config:

```bash
python3 -m venv ~/.buggy-venv
~/.buggy-venv/bin/pip install -e .

mkdir -p ~/.local/bin
ln -sf ~/.buggy-venv/bin/buggy     ~/.local/bin/buggy
ln -sf ~/.buggy-venv/bin/buggy-mcp ~/.local/bin/buggy-mcp
```

Check `~/.local/bin` is on your `PATH` (`echo $PATH`), then from any directory:

```bash
buggy functions src/pricing.py
buggy investigate apply_discount -f src/pricing.py \
  --pre 'price >= 0' --post 'result >= 0'
```

Or run it without installing anything:

```bash
PYTHONPATH=. python3 -m buggy.cli --help
PYTHONPATH=. python3 -m buggy.mcp          # the MCP server
```

The lack of dependencies is deliberate. The tool inspects your project, so it
must not be able to disturb your project's environment, and it should install on
a new Python release without waiting for anyone's wheels.

### Editor setup, in every project

An MCP server registered in a **workspace** config exists only for that folder.
Put it in the **user** config and every project gets it with no per-project
setup. For Kiro:

| File | Scope |
|---|---|
| `~/.kiro/settings/mcp.json` | every workspace - **use this one** |
| `<project>/.kiro/settings/mcp.json` | that project only, and it overrides the user config |

```json
{
  "mcpServers": {
    "buggy": {
      "command": "/Users/you/.local/bin/buggy-mcp",
      "args": [],
      "env": {},
      "disabled": false,
      "autoApprove": ["buggy_functions", "buggy_recall"]
    }
  }
}
```

An absolute path rather than a bare `buggy-mcp`, because an unrelated npm
package publishes commands with the same two names, and npm's global prefix
usually sits ahead of `~/.local/bin` on `PATH`.

`buggy_functions` and `buggy_recall` only read your code. The other four tools
prove a defect by importing the target module and executing it, which runs its
top-level code - so leaving them out of `autoApprove` means you are asked
first. More variants in
[`mcp-config.example.json`](mcp-config.example.json).

### If something tells you to install Node

It is out of date. Nothing here imports anything outside the standard library,
and there is no `package.json`. Check for yourself:

```bash
python3 -m unittest discover -s tests -t .
```

That suite passes in full with no Node installed. The TypeScript implementation
that did need it was removed; see [History](#history).

## Use

### Specifications are Python

Conditions are Python expressions. Your parameter names are in scope, and
postconditions also see `result`. The `math` module and the common builtins
(`len`, `abs`, `min`, `max`, `sum`, `all`, `any`, `isinstance`, `sorted`, ...) are
available.

```bash
buggy investigate apply_discount -f pricing.py \
  --pre  'math.isfinite(price) and price >= 0' \
  --pre  'percent_off >= 0' \
  --post 'result >= 0'
```

A condition that cannot be evaluated is **reported, never skipped silently**:

```
  ! postcondition not checked: !isNaN(result)
      not a Python expression (invalid syntax) - looks like JavaScript:
      use 'not x' instead of !x
```

Conditions run with no builtins and a restricted namespace. Lambdas, imports,
assignment expressions, dunder access and arbitrary attribute access are
rejected at compile time, so a specification cannot reach the filesystem or the
interpreter internals.

### Commands

| Command | What it does |
|---|---|
| `buggy functions <file>` | lists functions, their signatures, and which are investigable |
| `buggy investigate <name> -f <file>` | investigates one function |
| `buggy repair <name> -f <file>` | investigates, then proposes fixes and verifies each by execution |
| `buggy scan <file>` | investigates every supported function in a file |
| `buggy findings` | lists defects recorded in this project |
| `buggy recheck` | re-runs every recorded trigger and reports what changed |
| `buggy export-tests` | writes recorded defects out as pytest cases |
| `buggy-mcp` | runs as an MCP server over stdio, for use from an agent |

Useful flags: `--pre` / `--post` (repeatable), `--budget N` (inputs to execute,
default 300), `--timeout S` (per call, default 2), `--determinism N`,
`--json`, `--python PATH`.

### Methods

A qualified name reaches a class member: `buggy investigate Cart.total -f cart.py`.

Instance methods need an object to be called on. One is built from the class's
own `__init__` signature, trying candidate arguments until one constructs, and
**rebuilt fresh for every call** so state written by an earlier call cannot
change a later one. The certificate records the construction, because
`total(0)` is not something you can paste into a REPL and `Cart([1, 2]).total(0)`
is:

```
    built on    Cart([])  (rebuilt for every call)
    input       []
    broke       must not raise ZeroDivisionError

  Call it yourself to see it fail:
    Cart([]).average_price()
```

A method's real input is the pair *(how the object was built, what it was called
with)*, and both are searched. `Cart.average_price` takes no arguments at all;
its defect lives entirely in `Cart([])`, so varying only method arguments would
report it clean.

For the same reason, **preconditions can constrain constructor parameters** by
name:

```bash
buggy investigate Basket.scaled -f basket.py \
  --pre 'isinstance(items, list) and all(isinstance(v, float) for v in items)' \
  --pre 'math.isfinite(factor)'
```

Here `items` comes from `Basket.__init__` and `factor` from the method. On a name
clash the method's parameter wins.

When `__init__` takes only `*args`, or rejects everything tried, pass a receiver
yourself:

```bash
buggy investigate Session.retry -f session.py --receiver 'Session("localhost", 8080)'
```

A failure inside `__init__` is reported as a setup problem, never as a defect in
the method under test.

Functions in a package are loaded with their package context, so a module whose
first line is `from .rates import rate` works. This is not a detail: without it,
140 of the 149 analysable functions in buggy's own source reported "the module
failed to load", while every fixture passed because fixtures are standalone
files. Required keyword-only parameters are supplied by name for the same
reason, since omitting them raised `missing a required keyword-only argument`,
a harness artifact indistinguishable from a defect.

Still out of reach, each refused with a reason rather than silently reported
clean: special methods (`__init__`, `__add__`, ...), properties, `async` functions,
functions nested inside other functions, and `typing.overload` signatures,
which are not the implementation that runs. On the stdlib's `statistics`
module, 54 of 81 functions are investigable.

### Measured on real projects

Run over 34 installed packages and stdlib modules, about 243,000 lines of
unfamiliar pure Python: click, jinja2, attrs, packaging, pygments, pyparsing,
dateutil, idna, urllib3, requests, charset-normalizer, markupsafe, and stdlib
modules including `statistics`, `argparse`, `dataclasses`, `ipaddress`,
`difflib`, `configparser`, `urllib.parse` and `ast`.

| | |
|---|---|
| functions investigated | 3,651 |
| wall clock | ~6 min |
| crashes in buggy itself | **0** |
| calls exceeding 6 s | 4 |
| modules that would not load | 124 (3%) |

Every one of those 124 is a real reason, not a limitation: a module that is
Windows-only (`click._winconsole` asserts `sys.platform == "win32"`), or one
needing an optional dependency that is not installed (`OpenSSL`, `socks`,
`h2`). Their functions are reported unanalysable rather than clean.

Two limitations the runs show plainly. Receiver construction fails for classes
with demanding constructors, such as click's `Context` and pyparsing's
`ParserElement`, where `--receiver` is the answer. And on code with no
specifications most findings are weak by design, roughly two to one: a function
rejecting bad input, an argument of the wrong type, an interactive function
meeting an empty stdin. Preconditions are what turn that into signal.

Every fix in `tests/test_realcode.py` came from one of these runs, and none of
them could have come from a fixture.

### Programmatic

```python
from buggy.prove import Config, investigate

report = investigate(
    "budget.py", "budget_usage",
    preconditions=["budget >= 0"],
    postconditions=["result >= 0"],
    config=Config(max_executions=500, timeout=1.0),
)

if report.status == "proven":
    cert = report.certificate
    print(f"{cert.function}{tuple(cert.args)} broke {cert.rule}")

for warning in report.warnings:       # specifications that did not run
    print(warning.condition, warning.reason)
```

`investigate_file` investigates a whole module while reusing one worker.

### Repair

`buggy repair` proposes fixes for a proven failure and then **tests** each one.

```
  VERIFIED  return 0.0 when budget == 0
      guards the proven trigger (budget zero)
      fixes the trigger, 37 working inputs unchanged, 2/2 similar inputs also fixed
            BUG: `budget == 0` raises ZeroDivisionError.
            """
      +     if budget == 0:
      +         return 0.0
            return (spent / budget) * 100
```

Every candidate has to earn its verdict against four questions, which is
affordable because a call costs ~0.015 ms:

1. **Does it parse?** A patch that breaks the file is discarded.
2. **Does it fix the proven trigger?** The exact input from the certificate is
   re-run. The test is that the *proven* failure is gone, not that nothing is
   raised, so converting a `ZeroDivisionError` into a documented `ValueError`
   counts as a fix.
3. **Does it break anything that worked?** Every input the original handled
   correctly is re-run, and a changed result is a regression. "Handled
   correctly" excludes inputs that returned a value violating a stated
   postcondition, and inputs your preconditions exclude: a patch must not be
   blamed for changing behaviour on a call you said would never happen.
4. **Does it generalise?** Inputs sharing the trigger's shape (the same zero
   denominator with different numerators, other empty collections) are
   generated, screened to keep only those that fail the same way in the
   original, and must be handled too.

Question 4 is the honest overfitting check. A patch that special-cases the one
input that caught it passes 1-3 and fails here, on evidence rather than on a
coefficient. The verdicts are `verified`, `overfit` and `rejected`, each shown
with the reason:

```
  REJECTED  always return 0.0
      changes behaviour on [5, 2]: was ok:2.5, now ok:0.0 (37 of 38 inputs affected)

  OVERFIT   special-case exactly the triggering input
      only handles the exact input that caught it (0 of 4 similar inputs fixed)
```

The strategies are driven by the certificate, not by pattern-matching the
source: an early-return guard, a `raise`, a clamp for a magnitude problem such
as a discount above 100%, and a `None` substitution. For an instance method the
guard reaches through `self`.

**Nothing is ever written to your file.** Patches are returned as data.

### Remembering what was found

A certificate names an exact call that failed, which means it can be run again.
So instead of keeping notes, `buggy` keeps the triggers and re-executes them:

```bash
buggy investigate budget_usage -f budget.py --record   # store the certificate
buggy recheck                                          # re-run every trigger
buggy export-tests -o tests/test_regressions.py        # emit them as pytest
```

`recheck` reports one of four outcomes per finding, each from an actual
execution rather than from history:

| outcome | meaning |
|---|---|
| `still_failing` | the defect is still there |
| `fixed` | it was observed to pass |
| `regressed` | it had been observed fixed and is failing again |
| `unreachable` | the file or function is gone, so the trigger cannot be run |

`regressed` is the outcome worth wiring into CI, and `recheck` exits non-zero
when it finds one. It is **sticky**: a finding stays regressed until it actually
passes again, because downgrading it on the next run would let a build go green
while the defect is still there. `unreachable` is reported rather than counted
as fixed, since a deleted function is not a repaired one.

`export-tests` turns findings into ordinary pytest cases that assert the failure
is *gone*, so the file passes once the defects are fixed:

```python
def test_budget_usage_crash_b0b907():
    """must not raise ZeroDivisionError

    Proven 2026-10-30. Trigger: budget_usage(0, 0)
    """
    try:
        budget_usage(0, 0)
    except ZeroDivisionError as exc:
        pytest.fail(f"budget_usage still raises ZeroDivisionError: {exc}")
```

Asserting the fix rather than the bug is deliberate: a test asserting the defect
still exists has to be deleted when it is fixed, which is how a regression suite
rots.

Recording is **opt-in** (`--record`), and the database lives at
`.buggy/findings.db` in the project. A tool that inspects your code should not
start writing files to it uninvited. Add `.buggy/` to your `.gitignore`, or
commit it to share findings with the team.

### As an MCP server

`buggy-mcp` speaks MCP over stdio, so an agent can prove and repair defects
while it works. Copy an entry from
[`mcp-config.example.json`](mcp-config.example.json); for Kiro that goes in
`.kiro/settings/mcp.json`.

```json
{
  "mcpServers": {
    "buggy": {
      "command": "buggy-mcp",
      "autoApprove": ["buggy_functions", "buggy_investigate", "buggy_scan"]
    }
  }
}
```

Six tools: `buggy_functions`, `buggy_investigate`, `buggy_repair`,
`buggy_scan`, `buggy_recall` and `buggy_recheck`, mirroring the commands
above. `buggy_recall` is the one to reach for before editing a risky
function: it reports defects already proven against it and whether any
came back. Each returns both human-readable
text and `structuredContent`, and a certificate always carries a `reproduce`
field holding the exact call to paste into a REPL.

`buggy_repair` is deliberately **not** in the `autoApprove` list above. It
writes nothing, but a patch is a change to your code and deserves a human
glance.

Implemented against the standard library rather than the official MCP SDK. The
SDK installs cleanly, but brings 27 transitive packages including a web server
and a crypto library, which is disproportionate for a stdio pipe and would end
this package's dependency-free install. Supported methods are `initialize`,
`ping`, `tools/list`, `tools/call`, and empty `resources/list` and
`prompts/list`; protocol versions `2024-11-05` through `2026-07-28` are
negotiated, and an unrecognised one is answered with a supported version as the
spec requires.

## What "proven" means

There is no solver and no symbolic reasoning. A certificate is a record of
observed failure, and it is worth being precise about what that buys you:

- the input was **executed**, not inferred
- it satisfied every stated **precondition** (admissibility)
- the failure was **observed again** in a fresh process (soundness)
- it **recurred** in at least 2 of 3 further attempts (reproducibility)

So a proven finding is a true positive you can reproduce by hand. The other
statuses are kept distinct on purpose:

| status | meaning |
|---|---|
| `proven` | a failing input survived all three pillars |
| `unconfirmed` | the function ran and nothing failed. **Not** a proof of safety |
| `no_valid_input` | the preconditions rejected every candidate, so the function never ran. Nothing was learned |
| `skipped` | it could not be analysed or called at all; the reason is given |

`no_valid_input` exists because collapsing it into `unconfirmed` would present
an absence of evidence as evidence of absence. It was found by running this
against the stdlib: `statistics.fmean` under a reasonable precondition executed
zero inputs and still claimed nothing was wrong.

## What is checked

Five oracles, in this order. Your own postconditions are checked before the
built-in numeric ones, so a broken rule is reported in your words.

1. **timeout** - the call did not finish
2. **crash** - it raised, and the exception is the function's fault
3. **postcondition** - it returned a value you said was impossible
4. **non-finite** - it returned NaN or an infinity *from finite arguments*
5. **nondeterminism** - the same input gave different answers (`--determinism N`)

Three rankings keep the signal usable, because the easiest failures to trigger
are usually the least interesting:

**Propagation is not a defect.** `monthly(nan) -> nan` is IEEE 754 working, not a
bug. A non-finite result is only reported when the arguments were all finite, so
`ratio(0, 0) -> nan` is a finding and `monthly(nan)` is not. Without this, every
unguarded arithmetic function in a codebase gets flagged the moment the fuzzer
tries NaN.

**Type confusion is ranked last.** `divide(None, None)` raises `TypeError`, which
is trivial to trigger and is the caller's mistake. Such a finding is remembered
but the search continues, so `divide` is reported at `(x, 0)` with
`ZeroDivisionError` rather than at `(None, None)`. If type confusion is the only
failure found it is still reported, with advice to constrain the parameter types.

**A deliberate raise is a contract, not a defect.** `parse('oops')` raising
`SourceSyntaxError` is the documented behaviour, and on real code those findings
outnumbered everything else. An exception is treated as intended when the
function raises it explicitly, when an `@raises`/`:raises:` tag or a `Raises:`
section declares it, or when it is an exception class the module itself defines
(which covers propagating a module's own error from a helper).

Only structured markup counts, never prose. An earlier version read a bare
"raises X" from anywhere in the docstring and promptly suppressed a real defect
documented as `"""BUG: denominator == 0 raises ZeroDivisionError."""`.
Documenting a bug is the opposite of declaring a contract, and silently ignoring
that finding would have been far worse than the noise the rule removes.

## History

This started as a TypeScript tool. That implementation is gone from the working
tree but preserved intact at the `ts-final` tag:

```bash
git show ts-final --stat          # what it contained
git checkout ts-final             # the whole thing
git checkout ts-final -- src/     # one path out of it
```

It was removed rather than maintained alongside this, because a root
`package.json` declaring `engines.node` is the first thing any reader finds, and
it kept sending people to a Node install they do not need.

The TypeScript tool did execute Python: it shipped a `PythonExecutor`, a
`tree-sitter-python` grammar, Python-aware signature extraction, and a complete
Python repair dialect. The problems were narrower and worse than "it has no
Python support".

**Specifications were evaluated as JavaScript, whatever the target language.**
`spec-conditions.ts` compiled conditions with `new Function`, and all three call
sites caught the failure and continued. So a Python condition raised a JavaScript
`SyntaxError`, was swallowed, and the investigation reported "no bug found" -
indistinguishable from genuine safety. Its own example runner, still in
[`examples/checkout-service/run-buggy.mjs`](examples/checkout-service/run-buggy.mjs),
documents the workaround in a comment: *"Both are JavaScript expressions even
for Python targets."* Here, conditions are Python and an unevaluable one is a
warning.

**Execution cost made it unusable at size.** The original wrote four temp files
and spawned a fresh interpreter per input. Measured on this machine:

| approach | ms per execution |
|---|---|
| four temp files + fresh interpreter | 34.9 |
| persistent worker (this package) | 0.015 |

The committed example logs show the consequence: 494 seconds to investigate
seven trivial functions, with single functions taking 173 seconds.

**Three things are simpler in Python, not merely different.** `ast` replaces a
native grammar that could be configured into the wrong language and silently
fall back to parsing Python as TypeScript. `json` carries `NaN`, `Infinity`,
`-0.0` and big integers between two Python processes losslessly, so the
original's sentinel-encoding scheme is unnecessary in both directions. And the
~250-line regex TypeScript stripper has nothing to strip.

### Measured against the recorded run

[`examples/expense-api/logs/investigations.jsonl`](examples/expense-api/logs/investigations.jsonl)
records a real TypeScript run over the same seven functions. Same files,
conditions translated to Python:

| | TypeScript (recorded) | this package |
|---|---|---|
| defects proven | 3 of 7 | **4 of 7** |
| wall clock | 494 s | **0.4 s** |

The extra defect is `apply_discount`, where a discount above 100% returns a
negative price. It is real and documented in the example's own `WORKFLOW.md`;
the recorded run missed it for want of an input above 100. `tests/test_prove.py`
keeps those statuses as a regression baseline.

## What exists today

Wired into the live pipeline and tested:

- `ast`-based function discovery and signature recovery, including qualified
  names for class members, positional-only and keyword-only parameters,
  variadics excluded from arity, and `self`/`cls` correctly not counted
- instance methods, with a receiver constructed from `__init__`, rebuilt per
  call, overridable with `--receiver`, and recorded in the certificate;
  constructor arguments are searched and can be constrained by preconditions
- the Python specification language, with a restricted evaluator and the
  three-way verdict (holds / broken / cannot be evaluated)
- the persistent worker: reused process, protocol isolated from the target's
  stdout, timeout with kill-and-respawn, `RLIMIT_CPU` and `RLIMIT_FSIZE`
- annotation-driven input generation with Python's hazards, and the one-hot /
  all-edge / random schedule for wide signatures
- the five oracles, with propagation and type-confusion ranking
- three-pillar verification and certificates
- repair: certificate-driven patch strategies, each verified by execution
  against the four questions above, with regression and overfitting detection
  grounded in observed behaviour rather than a score
- the `functions`, `investigate`, `repair`, `scan`, `findings`, `recheck` and
  `export-tests` commands, with `--json`
- the MCP server: six tools over stdio, stdlib-only, with stdout isolated from
  the protocol so a print in your code cannot break a session

Not written yet, and not implied anywhere in the output:

- **a cross-project memory.** The TypeScript implementation promoted lessons to
  a sanitised index in your home directory, shared across every project. Left
  out: it writes outside the project for a weak signal, and a trigger from
  another codebase cannot be re-run here, so none of its claims could be
  checked.
- **generating hooks or steering files from defect history.** That is config
  synthesised from thin evidence. `buggy export-tests` serves the same wish
  with something specific and checkable.
- **resources and prompts** over MCP. `resources/list` and `prompts/list`
  answer with empty lists so clients that probe them do not error, but nothing
  is exposed that way.
- **async functions** - refused with a reason; the worker has no event loop.
- **special methods** - `__init__`, `__add__` and the rest are refused; their
  calling conventions vary too much to guess at.
- **defects that need several calls on one object** - a receiver is rebuilt for
  every call, which is what makes reproducibility sound, but it means a leak or
  a corruption that only appears on the third call is invisible.
- **a hard memory cap** - requested and *reported*, not assumed. `RLIMIT_AS`
  cannot be lowered on macOS (`setrlimit` raises), so `Executor.limits` tells
  you what the OS actually accepted rather than claiming a limit that is not
  there. `RLIMIT_CPU` does work, and is what stops a runaway loop.

Isolation today is a separate process with a CPU cap, a file-size cap, and
kill-on-timeout. There is no filesystem, network or syscall restriction. Do not
point this at code you would not run.

**Side effects happen.** Your functions are really executed, and for an instance
method so is the constructor, with generated arguments. Running buggy over its
own source left a stray SQLite file behind, because a generated argument reached
a constructor that opens a database. Nothing is faked or mocked, which is what
makes a finding reproducible and also what makes this a real consideration. Run
it on a clean working tree, and prefer `--receiver` to pin the construction when
a class touches the outside world.

The target's stdin and stdout are both pointed at `/dev/null`, so a `print`
goes nowhere and an `input()` gets `EOFError` immediately rather than waiting.
That keeps the protocol safe and stops an interactive function from looking
like a hang, but it is not a sandbox: a function that spawns an editor or opens
a URL will still do it.

## Tests

```bash
python3 -m unittest discover -s tests -t .    # no installs needed
# or
pytest
```

233 tests, all passing, about 39 seconds. `tests/test_realcode.py` holds the
regressions that only a run over real code could find, and is worth reading
before trusting the fixture suite. `tests/test_prove.py` includes the
parity baseline against the recorded TypeScript run and is skipped if the
example sources are absent. `tests/test_repair.py` injects a deliberately
sabotaged patch and a deliberately overfit one, to check that the evidence
actually rejects them.

## License

MIT

TDQS

A3.9/5.0

Scored across 6 tools

Disambiguation4/5

Each tool has a recognizably distinct role (list functions, recall history, investigate one function, repair, recheck, bulk scan), and the descriptions explicitly disambiguate the two closest pairs: investigate vs scan, and recall vs recheck. The remaining overlap is mild but real: scan is essentially a batched investigate, and recall and recheck both surface prior defect state, so an agent may hesitate between them in edge cases.

Naming Consistency4/5

All six tools share a consistent 'buggy_' namespace prefix followed by a short, predictable verb or noun. The only deviation is buggy_functions, which is a noun where every other tool uses a verb-style label, but it remains readable and idiomatic for a discovery tool.

Tool Count5/5

Six tools is tightly scoped to a single, well-defined workflow (discover, recall, prove, repair, recheck, triage). Every tool corresponds to a distinct stage with no filler, and the surface is small enough that an agent can hold the whole model in mind.

Completeness4/5

The lifecycle is nearly fully covered: discovery, historical recall, proof, patch proposal, regression recheck, and bulk triage all exist. The deliberate exclusion of file writing (repair only proposes) and the lack of any tool to manage or clear stored triggers/records are the only notable gaps, both workable around.

Maintenance

ActivityActive
ResponsivenessOutdated data