bleurs

Ground truth for AI coding agents.

Blocks the APIs that don't exist. Serves the ones that do. One index, both directions.

bleurs is a static verifier and API projector for Python and TypeScript that runs inside the edit path of an AI coding agent. Before a write reaches disk, it resolves every external reference in the proposed file against the environment that code will actually run in — the standard library, installed distributions, live library objects, PyPI and npm, node_modules, and your own project's symbol table — and rejects the edit if any reference is provably fictional. The same index answers the inverse question, projecting exact API surfaces so an agent can learn how to call something without reading the file that implements it.

Results

Measured on 200 real files from site-packages, each rooted at its own package so every resolution tier is active. Reproducible with two commands.

The third row is the one that matters. bleurs never reached a wrong verdict on a planted hallucination — it either caught it or said out loud that it was not judging. See Evidence for methodology and how to reproduce.

Across both languages that is 1,246 of 1,335 planted hallucinations caught (93.3%), zero false positives across 900 files, and nothing judged wrongly.

TypeScript scores higher on recall (99.4% against 83.5%) for an unglamorous reason: it claims less. Node's module resolution is a filesystem walk with deterministic fallbacks, while Python's surface is dynamic. The TypeScript front-end also does not implement class shapes or instance attributes at all — and every single bug the Python sweep uncovered lived in exactly that tier.

Related MCP server: mcp-public-apis

Contents

• Installation · Quick start · What it detects

• Evidence · How it works · Reference

• Comparison · FAQ · Limitations

• Roadmap · Prior art · Contributing

Installation

uv tool install git+https://github.com/Anandb71/Bleurs            # Python
uv tool install "bleurs[typescript] @ git+https://github.com/Anandb71/Bleurs"

The core has zero runtime dependencies: nothing to provision, no index to build, no database. It reads the interpreter's own ast module and its own packaging metadata.

TypeScript support adds tree-sitter, kept as an extra so a Python-only user never pays for grammars they will not load.

PyPI release pending. Once published this becomes uv tool install bleurs.

Try it without installing:

uvx --from git+https://github.com/Anandb71/Bleurs bleurs demo

Quick start

As an agent hook (the primary use)

Checking after the fact is a linter. Checking before the write is a firewall.

bleurs install-hook

Registers a PreToolUse hook on Write, Edit, and MultiEdit. bleurs reconstructs what the file would contain after the edit, verifies it, and rejects the tool call if any reference is provably fictional — returning the real API of whatever the agent got wrong:

Bleurs blocked this edit. These references do not exist:
  - t.py:5  base64.encode_string - base64 has no attribute 'encode_string'
  - t.py:5  json.loads_safe - json has no attribute 'loads_safe'

What those containers actually provide:

base64  [module]
  b64encode(s, altchars=None)
  b64decode(s, altchars=None, validate=False)
  urlsafe_b64encode(s)
  ...

json  [module]
  dumps(obj, *, skipkeys=False, ensure_ascii=True, ...)
  loads(s, *, cls=None, object_hook=None, ...)
  ...

The agent corrects on the next turn because it now has the ground truth, and it never opened a file to get it. Works with any client that can run a command on a JSON payload — Cursor, Codex, a custom loop. It reads a tool call on stdin and answers with an exit code.

As an MCP server

claude mcp add bleurs -- bleurs mcp

Exposes surface (what exists) and verify (what doesn't). See MCP tools.

On the command line

bleurs check src --exclude demo          # exit 1 if anything is blocked
bleurs surface datetime.datetime         # exact API, no file read
bleurs context src/app/service.py --stats  # everything needed to edit that file

# working set for hook.py
# 2803 of 3000 tokens, 14 modules
...
~2803 tokens across 14 modules; reading the equivalent files costs ~31693 — 11.3x

In CI

- run: pip install git+https://github.com/Anandb71/Bleurs
- run: bleurs check src

What it detects

The bolded row is where hallucinations in a real repository concentrate. An agent rarely invents a standard library function; it invents a method on the class you just showed it, and tools that only check imports pass straight over that.

Evidence

Two harnesses, both in benchmark/, both reproducible.

Precision

python benchmark/eval_hallucinations.py --limit 200

Runs bleurs over unmutated files from site-packages. Those files are installed, importable and working, so every reference in them resolves by construction. Any block is therefore a false positive — no labelling step, no judgement call, no opportunity to grade our own homework.

PRECISION  (unmutated working code; any block is a false positive)
  files checked        200
  false positives      0
  false positive rate  0.000%

This number has been 0 exactly twice, and both times it took work to get there. The first run of this harness reported 20%; five real bugs, all fixed (64dbf1c).

Then a worse problem surfaced — in the harness itself. It built its engine without a project_root, which silently disabled tier 0, so class shapes, self resolution and instance-attribute checking were never measured at all and a 0% rate was being published for a configuration nobody runs. Rooting each file at its own package turned the tier on and the rate went to 12%: nine further false-positive classes, from @staticmethod receivers to mixins to a project warnings.py shadowing the stdlib. All nine are fixed and pinned in tests/test_regressions.py, one test per bug with the library that produced it named (f7ec0fa).

The lesson is worth stating plainly: a benchmark that does not exercise the feature you are claiming for measures nothing, however good its number looks.

Recall

The same real files, with one reference deliberately broken — a method renamed to something plausible that does not exist, or a package swapped for one nobody published. Ground truth is exact because we know what was broken and where.

RECALL  (planted hallucinations, each verified absent before counting)
                          caught  declined  silent
  invented API               100        69       0
  invented import name       181         9       0
  invented package           143         6       0
  overall                    424        84       0    83.5% all / 100.0% judged

  why bleurs declined to judge:
      62  reference is inside a try/except that handles it failing
      11  resolves to project-local code we could not index
      11  dropped before judging (wildcard import, shadowed name, ...)

Three methodological commitments:

1. Every planted name is verified absent before it is scored. os_toolkit and argparse_utils are both real PyPI projects; a mutation that lands on something published is not a hallucination and is discarded.

2. Outcomes are three-way. A hallucination bleurs declined to judge is a miss, but a principled one, produced by the same rules that hold the false positive rate at zero. Folding those in with genuine blind spots would hide where the blind spots are.

3. silent is the honest column — cases examined and got wrong. It is zero, and that is the claim worth making, not the 83.5%.

Recall was 56.5% earlier in this project's life. Two changes moved it, and neither was a loosened threshold:

__getattr__ was being treated as a fog rather than a function. A module defining PEP 562 __getattr__ abstained outright — but typing defines one and still raises AttributeError for a name it does not have. That is the module itself answering, with more authority than any listing we could assemble. Asking it instead of guessing about it moved recall to 75.4%.

Introspection sees one operating system. signal.SIGQUIT is real on Unix and absent on Windows, so it had to abstain alongside signal.register_all, which is real nowhere. CI runs three platforms across four Python versions, so all twelve jobs now report what they see and the union ships as a 52 KB table. A name absent from every one of them exists nowhere. That took it to 83.5%.

The table is only ever allowed to permit: no entry, or an interpreter outside its range, means no opinion and the caller keeps abstaining. A stale table costs recall and can never cause a false block.

TypeScript

npm install react express lodash zod axios date-fns chalk rxjs   # a corpus
python benchmark/eval_typescript.py --corpus ./tscorpus --limit 700

PRECISION  (unmutated working code; any block is a false positive)
  files checked        700   (skipped 0, did not parse)
  false positives      0
  false positive rate  0.000%

  blocks Node also rejects (2) -- real package defects, not ours:
    * _lib/test.cjs:6 require("./test/vitest") -- no module at './test/vitest'

RECALL  (planted hallucinations, package names verified absent)
                            caught  declined  silent
  invented named import        252         5       0    98.1%
  invented package              15         0       0   100.0%
  missing relative module      555         0       0   100.0%
  overall                      822         5       0    99.4% all / 100.0% judged

Two methodological points the first run forced, both of which had been scoring the corpus rather than the checker:

Node adjudicates disputed blocks. The first run reported two false positives. Both were real: date-fns ships _lib/test.cjs requiring ./test/vitest, a file it does not publish, and require.resolve returns MODULE_NOT_FOUND. Asking Node rather than our own resolver keeps the measurement from being circular — and bleurs found a genuine broken import in a package with millions of weekly downloads.

A mutation only counts if it lands in code. date-fns documents its API with import examples inside JSDoc comments. The first version planted hallucinations there and then marked the checker wrong for correctly ignoring a comment.

Relative-path mutations are realistic typos — a dropped plural, a doubled letter, two adjacent characters swapped — each verified unresolvable by Node before being scored.

Still unmeasured here: tsconfig path aliases and baseUrl (unit-tested, but the corpus has no tsconfig), and monorepo workspaces.

Context reduction

python benchmark/surface_savings.py

393 files from site-packages, deliberately not this repository, whose comment density would flatter the result.

Token counts are estimated at 4 chars/token rather than measured with a real tokenizer, since shipping one would mean shipping a dependency. The estimate applies identically to both sides of every ratio, so it cancels.

How it works

The decision rule

BLOCK requires positive evidence of absence.

Not "we couldn't find it." Absence, demonstrated by a named resolver that looked inside a container it successfully opened.

flowchart LR
    A["reference in the<br/>proposed edit"] --> B["resolve against<br/>the tiers"]
    B --> C{outcome}
    C -->|"found"| D["ALLOW"]
    C -->|"container opened,<br/>name not in it"| E["BLOCK"]
    C -->|"could not open<br/>the container"| F["ALLOW<br/><i>and record why</i>"]

    style D fill:#1a7f37,stroke:#1a7f37,color:#fff
    style E fill:#b62324,stroke:#b62324,color:#fff
    style F fill:#7d4e00,stroke:#7d4e00,color:#fff

The third branch is the one most tools collapse into the second. The asymmetry is deliberate: a missed hallucination costs one failed test run you were going to have anyway, while a false positive costs the tool — it gets disabled, and from that moment catches nothing. Recall is worth spending; precision is not.

Resolution tiers

Cheapest and most certain first. Each answers present, absent, or unknown; only the middle answer can block.

For TypeScript and JavaScript the same discipline runs over different ground truth, because Node has no importlib.metadata and no runtime introspection:

A tsconfig paths alias is indistinguishable from a bare package by shape alone, so anything matching an alias prefix — or any bare specifier in a project that sets baseUrl — abstains rather than risking a false positive on entirely ordinary code.

Tiers 0 and 3 also power surface. Projecting an API and proving one absent are the same operation read in opposite directions, which is why both halves of this tool are one index rather than two systems sharing a repository.

Tier 3 is the only one that executes third-party code, because there is no other way to ask an object what it contains. It runs in a subprocess launched with -I, with a timeout, batched once per check rather than once per reference, and only for packages already installed in your environment. --no-introspect disables it, after which bleurs reports that it verified no APIs rather than implying the file is clean.

Class shapes

Tier 0 models each class as a shape: its methods, class attributes, attributes assigned onto self, and its base classes, with re-exports followed across files.

A shape is closed when the complete attribute surface can be enumerated. Only a closed shape may produce a block. A shape opens — and abstains — if any base cannot be resolved, an unrecognized decorator might have replaced the class, __getattr__ is defined, or attributes are set dynamically.

Types are read, never inferred. A variable is bound to a class only when it is assigned exactly once, by a plain assignment, from a bare constructor call:

user = User("a@b.c")        # bound   -> user.emial blocks
user = make_user()          # unbound -> abstains (factory return type unknown)
for user in load():         # unbound -> abstains
def f(user): ...            # unbound -> abstains

What it deliberately does not judge

bleurs check --explain reports which of these applied.

Reference

Commands

bleurs check

bleurs context

bleurs surface

MCP tools

Performance

Registry answers are cached on disk, so a package name is fetched once, ever. Introspection is batched per check, not per reference.

Comparison

Type checkers cover much of this column, and if you run pyright in strict mode you have a lot of it. Two things they do not do.

They cannot distinguish "not installed" from "never existed." mypy reports Cannot find implementation or library stub for module named 'X' whether X is a forgotten dependency or a name no human has published. That distinction is the entire security story: one is a missing install, the other is an open supply-chain hole waiting for someone to register it.

They run after the file exists. bleurs checks the proposed content of an edit that has not happened, which is the only point at which it can be stopped.

FAQ

Do — they compose well. Use pyright for types and bleurs for grounding.

They require configuration, benefit enormously from annotations, need stubs for untyped dependencies, and run on files that exist. bleurs requires none of that and runs on the proposed edit. And a type checker cannot tell a forgotten dependency from an unpublished name.

For finding code they are good, and several search better than bleurs does.

Embeddings retrieve what is similar; this retrieves what is true. A vector index returns chunks ranked by a similarity score, and a chunk boundary can cut a signature in half. A surface is the complete public API of a container, derived from the runtime object or the parse tree, with no ranking step to be wrong about.

They do not close the loop. A graph server answers when asked. It cannot know the agent just wrote something false, because it is not in the write path. bleurs is, so the moment of failure is also the moment of retrieval — the one moment you can be certain the information is needed.

The sharpest objection to the design, and the mitigation is scope.

Tier 3 only imports packages already installed in your environment — code that was going to execute the moment you ran your program. It installs nothing, fetches nothing, and never imports a name it just learned about. Anything not installed is settled by metadata and the registry, with no execution at all.

It runs in a subprocess launched with -I (isolated mode) under a timeout, so a package that hangs, prints, or calls sys.exit on import cannot take the checker with it. --no-introspect disables the tier entirely.

~300 ms per edit, most of it Python startup, against agent turns measured in seconds and wrong-turn recovery measured in minutes. --offline --no-introspect runs in ~200 ms and still catches invented packages.

A missed hallucination is a normal miss; --explain usually reports that it abstained and why.

A false positive is the highest-priority bug class in this repository and is fixed before features. tests/test_no_false_positives.py is written first, kept first, and gates every pull request. The precision harness exists to find them before you do — and it has.

Limitations

• TypeScript claims less than Python. No class shapes, no instance attributes, no self resolution — which is why its numbers are better, not because the front-end is smarter.

• TypeScript checks packages and project files, not package APIs. Member access on an npm dependency abstains, because answering it means resolving .d.ts files, exports maps and declaration merging — tsc's job, and not worth reimplementing badly. Project-local exports and package existence are fully decidable and are checked.

• Go and Rust are not supported yet. The Analyzer interface is the extension point; see Roadmap.

• Not a semantic checker. It proves a symbol exists, not that it is used correctly. The published AST-validation work measuring this approach reports 0% correction on contextual mismatches; that boundary is real.

• Not a replacement for conversation compaction. It removes the dominant cause of context pressure — file reads — and makes forgetting cheap to recover from. Your chat history is still your chat history.

• Not a search engine. context takes a file you are about to edit, not a sentence describing intent, and surface takes a name. Neither will find the starting point for you — that is deliberate, since a similarity ranking is exactly the kind of guess the rest of this tool refuses to make. Pair it with a code-search tool.

• The alias table is finite. If an import name is not installed, not stdlib, not project-local, not in aliases.py, and has no PyPI project of that name, bleurs blocks it. The residual risk is a real package whose import name differs from its distribution name and is missing from that table. Run --no-strict-imports to downgrade registry-based blocks to warnings. Additions to that table are the highest-value one-line PR here.

• Platform-varying stdlib containers are not judged. A deliberate recall cost; see What it deliberately does not judge.

Roadmap

• .d.ts resolution for TypeScript, so member access on npm packages is decidable rather than abstained. That means exports maps and declaration merging, which argues for driving tsc rather than reimplementing it.

• Go and Rust front-ends behind the same Analyzer interface. The difficulty is never parsing; it is ground truth. Each language needs an answer to "what is installed" and "what does this expose".

• SCIP ingestion so tier 0 can consume an existing SCIP index rather than walking the filesystem, inheriting real cross-file name resolution.

• Grounded generation. Enumerate the permitted reference set and supply it to the model before it writes, rather than checking afterwards — the difference between a spell-checker and a keyboard with only real words on it.

Prior art

bleurs stands on a great deal of other people's work.

Contributing

See CONTRIBUTING.md. One rule: tests/test_no_false_positives.py must stay green. A change that catches more hallucinations but blames correct code is not an improvement.

Good first contributions: a missing pair in aliases.py, a false positive found in the wild, or a language front-end.

License

MIT © Anand Biju