Provena

Track where every line of AI-generated code came from.

When Claude (or any agent) writes code, Provena records what it saw (files read, pages fetched, your instructions) and what it produced, then lets you ask:

provena_why src/auth.ts:42
→ ← docs/oauth-spec.md [file] via declared (conf 1.00)
     evidence: "access tokens expire after 15 minutes"

Lines with no backing source are flagged ungrounded — model knowledge to verify by hand. That honesty is the point.

📄 Paper: PAPER.md — Hook-Mediated, Span-Level Provenance for Code Written by LLM Agents. Method, benchmarks, and results (0% false attribution across all configs; held-out F1 90.9–94.7%, ceiling 95.7–100%).

How it works

flowchart LR
  subgraph Session["Claude Code session"]
    R["Read / WebFetch / Grep"]
    P["your prompts"]
    W["Write / Edit"]
  end
  R -- PostToolUse hook --> SRC[(source)]
  P -- UserPromptSubmit hook --> SRC
  W -- PostToolUse hook --> ART[(artifact)]
  subgraph Store[".provena/provenance.db — local SQLite"]
    SRC
    ART
    EMB[(embedding cache)]
  end
  ART --> ENG{{"attribution engine"}}
  SRC --> ENG
  ENG -- "s ≥ HIGH" --> G["grounded (embedding)"]
  ENG -- "overlap band\n[floor, HIGH)" --> J["LLM judge\n(top-K, full source)"]
  ENG -- "s < floor" --> U["ungrounded"]
  J --> G
  J --> U
  G & U --> Q["provena why / audit / gate / export"]

Capture needs no cooperation from the model — hooks sit in the tool path. Storage is local SQLite (Node's built-in node:sqlite), so nothing leaves your machine. Embedding asserts in the confident region; the judge owns the overlap band where grounded and ungrounded similarities mix, which is what holds false attribution to 0%.

Related MCP server: Bicameral MCP

Requirements

• Node ≥ 23.6 (runs TypeScript directly; uses built-in node:sqlite)

Setup

Install from source (GitHub):

git clone https://github.com/parkseokjune/provena.git
cd provena
npm install            # installs @modelcontextprotocol/sdk + zod + local embeddings
node src/cli.ts init   # wires hooks into .claude/settings.json + registers MCP in .mcp.json

Run provena against your own project by passing its path, or copy the provena dir alongside your repo. Then restart Claude Code so it loads the hooks and the provena MCP server.

CLI

MCP tools

LLM judge (optional)

Embedding attribution decides the confident cases on its own. For the borderline band, an LLM judge reads the candidate sources and decides derivation. Set one key:

export GEMINI_API_KEY=...        # uses gemini-2.0-flash
# or
export ANTHROPIC_API_KEY=...     # uses claude-haiku-4-5
# optional overrides:
export PROVENA_JUDGE_MODEL=...        # pick a specific model
export PROVENA_JUDGE_PROVIDER=gemini  # force a provider if both keys are set

Without a key, borderline spans are honestly reported as uncertain rather than guessed.

Status

• Phase 0 ✅ attribution-accuracy spike (spike/) — 100% top-1 retrieval, ungrounded cleanly separated

• Phase 1 ✅ capture + store + MCP query

• Phase 2 ✅ embedding attribution engine + LLM judge + audit report + eval harness. Embedding-only F1 94.7%, 0% false-attribution.

• Phase 3 ✅ live judge (gemini-2.5-flash-lite) reaches F1 100% on Benchmark A; multi-language held-out Benchmark B (TS/Python/Go): embedding-only test F1 90.9%, 0% false-attribution, oracle ceiling 95.7%, live 90.9% (eval/RESULTS.md iter 3–4). CI gate (provena gate) shipped. Paper in PAPER.md.

• Next ⬜ larger naturally-occurring corpus, judge-model sweep, signed regulatory export.

Try the evaluation

node src/eval.ts        # labeled benchmark: precision / recall / F1 / false-attribution
node test-judge.ts      # judge wiring unit test (mocked transport)