# provenance-mcp

Answers "why does this code exist" for a `file:line` or a natural-language question in [colinhacks/zod](https://github.com/colinhacks/zod), by tracing code → git blame → commit → PR → linked issue. Every answer is either quoted from real history with a citation, or explicitly marked `NOT RECOVERABLE` -- never a plausible-sounding guess.

v1 is hardcoded to a single demo repo (`colinhacks/zod`). No multi-repo support, no config system.

## Architecture

- `packages/core` -- shared TypeScript retrieval logic: local git blame/history (including pickaxe search and full non-line-scoped file history), GitHub REST enrichment (PR/issue linkage, review comments, discussion threads, repo-scoped issue/PR search), reading current file content (README/docs/changelogs), hunk-based diff chunking, commit-importance scoring, Upstash Vector indexing/search, and Report/citation assembly. All tracing logic lives here exactly once.
- `packages/mcp-server` -- a thin [MCP](https://modelcontextprotocol.io) server (`@modelcontextprotocol/sdk`) exposing four tools: `trace_line`, `ask_why`, `search_history`, `investigate`. Calls into `core` only.
- `packages/demo` -- a small live web app comparing provenance-mcp's answers against raw `git log --grep`, a web-search-equipped Claude call, and provenance-mcp's own multi-hop mode, side by side. See [Demo](#demo) below.
- `packages/skill` -- `SKILL.md`, a Claude Code Skill that drives the MCP tools and renders the fixed `WHY / EVIDENCE / STILL LIVE? / VERDICT / CONFIDENCE` verdict block.
- `scripts/index-repo.ts` -- one-off script that clones zod locally, walks its full commit history, and upserts commit documents into Upstash Vector.

`trace_line`/`ask_why`/`search_history` never call an LLM -- core returns structured, cited JSON, and the *calling* agent (Claude, via the Skill's procedure) is the one that renders prose. `investigate` is the one exception: it runs a bounded, LLM-orchestrated multi-hop loop (see below), but the model only ever chooses *where to look next* -- the actual cited text is still deterministically re-extracted from real fetched sources by the same `evidence.ts` logic the other tools use. There's no code path where model-authored prose becomes a citation.

### The `investigate` tool (multi-hop mode)

`trace_line`/`ask_why` take a single deterministic pass: blame or vector-search to one anchor commit, follow its linked PR/issue, done. That's fast and cheap, but it can miss things a single pass can't reach -- a rationale stated in a review comment rather than the PR body, a fix that requires retrying the search with different words, or a design decision documented in current prose (a changelog, a migration guide) rather than any single commit.

`investigate` runs an autonomous loop (default: up to 40 turns / 60 tool calls / 4 minutes, Opus-tier model) with 11 read-only navigation tools -- `blame_line`, `get_commit`, `get_commit_diff`, `pickaxe_search`, `file_history`, `vector_search`, `get_issue_or_pr`, `get_pr_review_comments`, `get_discussion_comments`, `search_issues`, `read_doc` -- plus two terminal tools, `submit_finding` and `give_up`. Every navigation tool result carries a harness-minted `sourceId` tied to real fetched text; the model can only ever point at IDs it has actually seen. When it calls `submit_finding`, that's a *pointer*, not an answer -- the harness re-reads that exact source's real text through the same `findRationale`/`buildReason` logic the deterministic tools use, and if the model's chosen source doesn't actually state a reason, the report honestly comes back `partial`/`not_recoverable` regardless of what the model claimed.

Requires `ANTHROPIC_API_KEY`. Prefer `trace_line`/`ask_why` first; only reach for `investigate` when they come back `partial` or `not_recoverable`.

## Setup

```bash
npm install
cp .env.example .env
# fill in GITHUB_TOKEN, UPSTASH_VECTOR_REST_URL, UPSTASH_VECTOR_REST_TOKEN
# ANTHROPIC_API_KEY is optional -- only needed for `investigate` and the demo's
# Exhibits B/D. trace_line/ask_why/search_history work without it.
```

Create the Upstash Vector index with a built-in embedding model (e.g. `mxbai-embed-large-v1`) so text can be upserted/queried directly without a separate embeddings step.

Build all workspaces:

```bash
npm run build
```

Index the demo repo (one-off; clones `colinhacks/zod` into `./.cache/zod`, ~2,900 commits):

```bash
npm run index-repo
```

Run the MCP server (stdio transport):

```bash
npm run mcp-server
```

Point an MCP client (Claude Code, the MCP inspector, etc.) at `packages/mcp-server/dist/index.js`, and install `packages/skill/SKILL.md` as a Claude Code Skill to get the cited verdict format end to end.

## Tools

| Tool | Input | Output |
| --- | --- | --- |
| `trace_line` | `{ file, line, ref? }` | `Report` -- precise file:line trace |
| `ask_why` | `{ question }` | `Report`, with `candidates[]` if the match is ambiguous |
| `search_history` | `{ query, path?, topK?, context? }` | ranked `Citation[]` from the vector index |
| `investigate` | `{ file?, line?, question?, ref?, maxTurns? }` | `Report`, with `investigationTrail[]` showing the search path. Requires `ANTHROPIC_API_KEY`; provide either `file`+`line` or `question` |

See `packages/core/src/types.ts` for the full `Report`/`Citation` schema.

## Demo

```bash
npm run demo
```

Serves a live page at `http://localhost:5173` comparing four ways of answering the same question about `colinhacks/zod`:

- **Exhibit A** -- raw `git log --grep`, no synthesis: what a developer gets from blind keyword search.
- **Exhibit B** -- a live Claude call with the real `web_search` tool, but no access to this repo's own git history / provenance-mcp index. The fair steelman: general web search vs. specialized primary-source tracing.
- **Exhibit C** -- provenance-mcp's `ask_why` (the single-shot deterministic tool).
- **Exhibit D** -- provenance-mcp's `investigate` (multi-hop), auto-escalated to whenever C doesn't come back fully `recovered`.

The pre-loaded example questions are verified, not cherry-picked: one case where the fast tier alone beats web search outright, one where web search states something false with full confidence and only multi-hop corrects it, and one pulled from a real (at the time, unanswered) open GitHub issue. See the comments in `packages/demo/src/examples.ts` for exactly what was verified about each.

## Testing

```bash
npm test
```

Runs Node's built-in test runner (`node:test`, no extra dependency) over `packages/core`'s pure logic -- rationale extraction, confidence scoring, citation builders, commit-importance heuristics, diff chunking, and `investigate`'s caching/ordering helpers. Two of the test cases are direct regression tests for real bugs found and fixed during development (a keyword-relevance threshold bug in `findRationale`, and a cache-upgrade bug in `investigate`'s commit caching) -- see the comments in `packages/core/src/investigate.test.ts` for what they'd have caught.