HonestCode

The deterministic verification layer for AI coding agents.

Catch invented APIs, undefined symbols, wrong calls, and other code hallucinations before your agent moves on.

The problem

AI coding agents hallucinate. They invent function names, fabricate library APIs, and call methods that do not exist.

HonestCode is the layer that checks what the agent just wrote against the repository it is supposed to be grounded in.

Agent
  ↓
writes code
  ↓
HonestCode
  ↓
evidence
  ↓
Agent fixes
  ↓
verified code

No LLM calls. No network. Pure AST + symbol resolution.

Related MCP server: javalens-mcp

Demo

The agent was asked to "add authentication using the existing UserClient." It generated:

client.refresh_token()

The real project only has:

class UserClient:
    def refresh(self): ...
    def refresh_access_token(self): ...

HonestCode catches it deterministically:

✗ verification failed — 1 issue in app/login_broken.py

  [invented_api] app/login_broken.py:9
  `UserClient.refresh_token()` does not exist.
  did you mean: refresh_access_token()?
  available methods:
    - refresh()
    - refresh_access_token()
  confidence: deterministic · action: revise

Run it yourself:

cd demos/invented-api
python run.py

Why not just use Ruff / Pyright?

HonestCode is not a replacement — it is a verification layer that answers a specific question: did the agent's code actually come from this repository?

The key difference is repository grounding. Ruff checks the file. Pyright types the call. HonestCode checks whether the call resolves to a real symbol that exists in the codebase the agent is editing.

Install

pip install honestcode

Requires Python >= 3.10. 100% local.

30-second setup with Claude Code

# Add HonestCode as an MCP server
claude mcp add honestcode -- honestcode-mcp

Or add it manually to your Claude Code config (~/.claude/config.json on macOS/Linux, %LOCALAPPDATA%\Claude\config.json on Windows):

{
  "mcpServers": {
    "honestcode": {
      "command": "honestcode-mcp",
      "args": []
    }
  }
}

Restart Claude Code. You now have two verification tools available:

• scan_file(path) — legacy tool, returns issues plus the new evidence shape.

• verify_file(path) — agent-facing tool, returns status, findings with evidence, and a human-readable text summary.

Try prompting Claude:

Add a login endpoint using the existing UserClient, then run verify_file on the
file you just wrote and fix anything it reports.

HonestCode auto-discovers the project root from the file path, indexes the codebase, and returns grounded evidence.

Real example

# auth/client.py
class UserClient:
    def refresh_access_token(self): ...

# auth/login.py
from auth.client import UserClient

def login(client: UserClient):
    client.refresh_token()  # invented API

>>> from honestcode.mcp.tools import verify_file
>>> verify_file("auth/login.py")
{
  "status": "fail",
  "file": "auth/login.py",
  "findings": [
    {
      "line": 7,
      "kind": "invented_api",
      "symbol": "refresh_token",
      "owner": "UserClient",
      "message": "`UserClient.refresh_token()` does not exist.",
      "evidence": {
        "available_methods": ["refresh", "refresh_access_token"],
        "did_you_mean": "refresh_access_token"
      },
      "confidence": "deterministic",
      "action": "revise"
    }
  ],
  "text": "✗ verification failed — 1 issue in auth/login.py\n  ..."
}

Agent output protocol

verify_file returns evidence an agent can act on directly, not just an error message:

{
  "status": "fail",
  "file": "auth/client.py",
  "line": 42,
  "kind": "invented_api",
  "symbol": "refresh_token",
  "owner": "UserClient",
  "message": "UserClient.refresh_token() does not exist.",
  "evidence": {
    "available_methods": ["refresh", "refresh_access_token"]
  },
  "confidence": "deterministic",
  "action": "revise"
}

How it works

1. Auto-index the project (or reuse the cached symbol index).

2. Parse the target file with the standard-library ast module.

3. Resolve every call site to a concrete symbol:

   • infer the receiver type from annotations, constructors, and imports;

   • reconstruct the class's member surface from the repository AST;

   • mark the surface as unknown when a base class cannot be resolved or the class defines __getattr__.

4. Emit findings with kind, owner, evidence, confidence, and action.

The loop is deterministic and auditable.

MCP tools

By default only scan_file is exposed. Set HONESTCODE_TOOLS=all to enable the full set, including:

See the old tool reference below for the complete list.

Benchmark

HonestCode includes two benchmark suites that run automatically in CI on every push and pull request to main:

Agent-accuracy benchmark

benchmarks/agent_accuracy/ is a deterministic, LLM-free benchmark that measures how well HonestCode catches common agent hallucinations.

Each task is a tiny agent episode: the agent writes a broken file, HonestCode verifies it, then the file is replaced with the fix and verified again.

cd benchmarks/agent_accuracy
python run.py

Current results (4 tasks, deterministic verification):

Summary: precision 1.0, recall 1.0, F1 1.0, false-positive rate 0.0, median verify time 2.51 ms.

Performance benchmark

scripts/benchmark.py measures the latency of core operations (index, scan, graph, dead code detection, similarity search) on the repository itself.

python scripts/benchmark.py                          # text output
python scripts/benchmark.py --format markdown        # table for README
python scripts/benchmark.py --repo psf/requests      # benchmark a real-world repo

Both benchmarks run in CI (.github/workflows/ci.yml) as separate jobs: benchmark-accuracy and benchmark-performance. A benchmark failure blocks the build if precision or recall drops below 1.0.

See benchmarks/agent_accuracy/README.md for the dataset format and how to add new accuracy tasks.

Architecture

honestcode/
├── verify/        # Evidence protocol + repository-grounded verification
├── mcp/           # MCP server layer
├── honest/        # Symbol index + project binding
├── graph/         # Persistent call graph (SQLite)
├── structure/     # AST extraction
├── sandbox/       # Sandboxed execution
└── cli.py         # Command-line interface

verify_file is the agent interface. scan_file is the default MCP tool and returns both the new evidence shape and the legacy issues list.

Roadmap

v0.1 — Repository grounding (now)

• symbol / API / function / class / method / import / call verification

• invented API detection with structured evidence

• auto-index on scan_file

v0.2 — Semantic contract verification

• function expects UserID, agent passes User → suspicious

v0.3 — Execution verification

• static verification → tests → runtime evidence

v1.0 — Verification Runtime for Coding Agents

                 Coding Agent
                      │
              ┌───────▼───────┐
              │   HonestCode  │
              │ Verification  │
              │    Runtime    │
              └───────┬───────┘
                      │
       ┌──────────────┼──────────────┐
       ↓              ↓              ↓
   Static          Semantic       Runtime
 Verification     Verification   Verification
       │              │              │
       └──────────────┼──────────────┘
                      ↓
                   Evidence
                      ↓
                     Agent

Development

git clone https://github.com/Fengrru/honestcode.git
cd honestcode
python -m venv .venv
.venv\Scripts\activate  # Windows
# source .venv/bin/activate  # macOS/Linux
pip install -e ".[dev]"
pytest
ruff check honestcode tests scripts

Run benchmarks locally:

# Accuracy benchmark
python benchmarks/agent_accuracy/run.py

# Performance benchmark
python scripts/benchmark.py

See CONTRIBUTING.md for pull request guidelines.

Security

See SECURITY.md for vulnerability reporting.

Telemetry

HonestCode collects no telemetry. There are no analytics libraries, no background services, and no phone-home endpoints.

License

MIT - Copyright (c) 2026 HonestCode Team