# RepoZip AI

An **AI repository index** that lets an assistant reason about an entire codebase almost entirely from a compact, persisted index instead of re-reading source files.

RepoZip starts from an ultra-dense **AST symbol map** (exports, interfaces, types, classes, public methods, imports — bodies, comments, and private members stripped) and enriches it into a navigable index: per-file semantic responsibilities, a global symbol table, a bidirectional dependency graph, feature clusters, a Next.js routing map, a React component tree, a search index, and a change predictor.

The full index payload is gated behind a **Solana Devnet micro-payment** and exposed to Cursor through an **MCP server**. Lightweight navigation tools (`search_repo`, `predict_changes`) are free.

## Stack

- Next.js (App Router) + TypeScript
- TailwindCSS
- `@solana/kit` for Devnet payment verification
- `@modelcontextprotocol/sdk` for the MCP server
- `@babel/parser` + `@babel/traverse` for AST compression
- `react-force-graph-2d` for the symbol graph

## What it does (in one picture)

The problem: AI coding assistants burn huge token budgets re-reading source files on every
task. RepoZip reads the whole repo **once**, distills it into a dense, structured index, and
serves that index to the assistant on demand — so the AI reasons about the codebase without
opening files. Access to the heavy payload is metered by a **Solana Devnet micro-payment**.

```mermaid
flowchart LR
    Repo["Your repository<br/>(raw source)"] -->|"read ONCE"| Pipeline["Indexing pipeline<br/>(10 layers)"]
    Pipeline -->|"persist"| Store[".repozip/<br/>compact index"]
    Store --> MCP["MCP server"]
    MCP <-->|"tools"| AI["Cursor AI"]
    Pay["Solana Devnet<br/>micro-payment"] -.->|"unlocks paid tools"| MCP

    classDef repo fill:#1f2937,stroke:#4b5563,color:#f9fafb;
    classDef core fill:#4338ca,stroke:#6366f1,color:#f9fafb;
    classDef pay fill:#065f46,stroke:#10b981,color:#f9fafb;
    class Repo,Store repo;
    class Pipeline,MCP,AI core;
    class Pay pay;
```

## Architecture

### System overview

Three surfaces sit on top of one shared index: a **Next.js web app** (visual demo + payment
UI), a set of **HTTP API routes**, and a **stdio MCP server** that Cursor talks to. All of them
call into the same indexing library and Solana verifier.

```mermaid
flowchart TB
    subgraph Client["Browser"]
        UI["Next.js App Router UI<br/>src/app/page.tsx"]
        Wallet["Browser wallet<br/>(Phantom / Solana)"]
    end

    subgraph Server["Next.js server + Node"]
        subgraph API["HTTP API routes"]
            Compress["/api/compress"]
            Verify["/api/verify-tx"]
            Search["/api/search"]
            Predict["/api/predict"]
        end
        MCP["MCP server<br/>mcp-server.ts (stdio)"]
        subgraph Lib["Shared library (src/lib)"]
            Indexer["index/indexer.ts<br/>build / load / update"]
            Solana["solana.ts<br/>verify + replay guard"]
            StoreLib["index/store.ts<br/>.repozip persistence"]
        end
    end

    Chain["Solana Devnet<br/>RPC"]
    Cursor["Cursor AI agent"]

    UI --> Compress
    UI --> Search
    UI --> Predict
    UI --> Verify
    Wallet -->|"signed payment"| Chain
    Verify --> Solana
    MCP -->|"paid tools"| Solana
    Solana -->|"getTransaction"| Chain
    Compress --> Indexer
    Search --> Indexer
    Predict --> Indexer
    MCP --> Indexer
    Indexer --> StoreLib
    Cursor <-->|"MCP tools over stdio"| MCP

    classDef ui fill:#1f2937,stroke:#4b5563,color:#f9fafb;
    classDef core fill:#4338ca,stroke:#6366f1,color:#f9fafb;
    classDef pay fill:#065f46,stroke:#10b981,color:#f9fafb;
    class UI,Wallet,Cursor ui;
    class Compress,Verify,Search,Predict,MCP,Indexer,StoreLib core;
    class Chain,Solana pay;
```

### The indexing pipeline (10 layers)

`buildIndex(root)` scans and AST-compresses every file **once** (the expensive step), then
`assembleIndex()` derives every cross-file layer with cheap in-memory map operations. This is
why incremental updates are fast: only the changed file is re-parsed; the derived layers are
recomputed from cached per-file entries.

```mermaid
flowchart TB
    Start["Repository root"] --> Scan["fileScanner.scanRepo<br/>safe walk: ignore rules,<br/>size caps, traversal guards"]
    Scan --> L0["Layer 0 · astCompressor<br/>parse to AST, strip bodies/<br/>comments, keep public API"]

    L0 --> PerFile{{"per-file entry<br/>(buildFileEntry + classifyFile)"}}
    PerFile --> L1["Layer 1 · metadata<br/>language, moduleType, role"]
    PerFile --> L2["Layer 2 · semanticClassifier<br/>responsibilities + summary"]

    L1 --> Assemble
    L2 --> Assemble

    subgraph Assemble["assembleIndex · cross-file layers (cheap map ops)"]
        direction TB
        L3["Layer 3 · symbolTable<br/>global exported symbols"]
        L4["Layer 4 · dependencyGraph<br/>fwd + reverse, multi-kind edges"]
        L5["Layer 5 · featureClusterer<br/>feature map"]
        L6["Layer 6 · routingMapper<br/>Next.js routes"]
        L7["Layer 7 · componentTree<br/>React hierarchy"]
        L8["Layer 8 · searchIndex<br/>inverted index + BM25-lite"]
    end

    Assemble --> L9["Layer 9 · changePredictor<br/>task -> likely / possible / indirect"]
    L9 --> Persist["index/store<br/>persist to .repozip/"]

    classDef scan fill:#1f2937,stroke:#4b5563,color:#f9fafb;
    classDef layer fill:#4338ca,stroke:#6366f1,color:#f9fafb;
    classDef out fill:#065f46,stroke:#10b981,color:#f9fafb;
    class Start,Scan,PerFile scan;
    class L0,L1,L2,L3,L4,L5,L6,L7,L8,L9 layer;
    class Persist out;
```

### The workflow it enables (AI + MCP)

Once indexed, the assistant navigates with **free** lightweight tools and only pays when it
needs the full payload. No full repository rescan happens during normal editing.

```mermaid
sequenceDiagram
    participant U as User
    participant AI as Cursor AI
    participant MCP as MCP server
    participant IDX as Index store

    U->>AI: "Add OAuth"
    AI->>MCP: predict_changes("Add OAuth")  [free]
    MCP->>IDX: read change predictor
    IDX-->>MCP: likely: auth, login, middleware...
    MCP-->>AI: ranked files + risk + confidence
    AI->>MCP: search_repo("authentication")  [free]
    MCP-->>AI: most relevant files
    Note over AI: opens ONLY those files
    AI->>AI: make code changes
    AI->>MCP: update_file_index(path)  [free]
    MCP->>IDX: reparse 1 file, re-derive layers
    Note over AI,IDX: no full rescan
```

### Payment / paywall flow (Solana Devnet)

Paid tools (`get_repo_index`, `get_file_index`, `get_compressed_repo_context`) require a
verified, unused Devnet signature. Verification is fully server-side and replay-protected —
the client's claim of success is never trusted.

```mermaid
sequenceDiagram
    participant W as Wallet
    participant MCP as MCP server
    participant S as solana.ts
    participant RPC as Devnet RPC
    participant IDX as Index

    W->>RPC: pay >= MIN_LAMPORTS to MERCHANT_ADDRESS
    RPC-->>W: txHash (signature)
    W->>MCP: get_repo_index(txHash)
    MCP->>S: verifyPayment(txHash)
    S->>RPC: getTransaction(txHash)
    RPC-->>S: recipient, pre/post balances, status
    alt not verified (wrong recipient / amount / failed)
        S-->>MCP: verified=false
        MCP-->>W: PAYWALL_REQUIRED
    else already used
        MCP-->>W: PAYWALL_REQUIRED (replay)
    else verified & unused
        MCP->>S: consumeSignature(txHash)
        MCP->>IDX: loadOrBuildIndex
        IDX-->>MCP: tiered payload
        MCP-->>W: index payload (unlocked once)
    end
```

> In `REPOZIP_DEV_UNLOCK` mode the paywall is skipped entirely for local demos.

### Core modules

- `src/lib/fileScanner.ts` — Safe recursive repo walk (ignore rules, size caps, path-traversal guards).
- `src/lib/astCompressor.ts` — Parse to AST, strip bodies/comments, keep public API surface.
- `src/lib/tokenEstimator.ts` — Estimate original vs compressed tokens + savings.
- `src/lib/graphBuilder.ts` — Build `{nodes, links}` for visualization.
- `src/lib/solana.ts` — Devnet payment verification + replay protection.

### Index layers (`src/lib/index/`)

- `types.ts` — All index data structures (`FileIndexEntry`, `SymbolEntry`, `FeatureCluster`, `RouteEntry`, `ComponentNode`, `ChangePrediction`, `RepoIndex`, ...).
- `store.ts` — Persistence to `.repozip/` (per-file shards, manifest, search index) + content hashing.
- `metadata.ts` — Layer 1 per-file enrichment.
- `semanticClassifier.ts` — Layer 2 responsibilities (deterministic heuristics + optional LLM enrichment).
- `symbolTable.ts` — Layer 3 global symbol table with param/return parsing.
- `dependencyGraph.ts` — Layer 4 graph with reverse edges, tsconfig `@/` alias resolution, and `imports`/`extends`/`implements`/`renders`/`apiToService`/`hookToComponent` edges.
- `featureClusterer.ts` — Layer 5 feature clustering by responsibility + graph connectivity.
- `routingMapper.ts` — Layer 6 Next.js App Router mapping.
- `componentTree.ts` — Layer 7 React component hierarchy.
- `searchIndex.ts` — Layer 8 field-boosted inverted index + BM25-lite ranking.
- `changePredictor.ts` — Layer 9 change prediction (no fresh scan).
- `indexer.ts` — Orchestrator: `buildIndex`, `loadOrBuildIndex`, and incremental `updateFile`.
- `query.ts` — Tiered payload helpers (overview / files / symbols / file detail).

### On-disk index (`.repozip/`, gitignored)

- `manifest.json` — Repo-level derived layers + per-file hash list.
- `files/<slug>.<hash>.json` — One shard per file (its `FileIndexEntry`), for cheap incremental writes.
- `search.json` — Inverted index postings.

## MCP tools

Registered via [.cursor/mcp.json](.cursor/mcp.json).

- `get_repo_index` — Tiered index (`overview` | `files` | `symbols` | `full`). Paid. Start with `overview`.
- `get_file_index` — Full detail for one file (Tier 3). Paid.
- `get_compressed_repo_context` — Backward-compatible full AST map. Paid.
- `search_repo` — Rank files by a natural-language query. Free.
- `predict_changes` — Predict likely/possible/indirect files for a task, with risk + confidence. Free.
- `update_file_index` — Incrementally refresh one file's index after an edit. Free.

Paid tools require a verified Devnet `txHash`; without one they return `PAYWALL_REQUIRED` (skipped entirely in dev-unlock mode).

## Setup

```bash
npm install
cp .env.local.example .env.local   # fill in MERCHANT_ADDRESS
npm run dev
```

Open http://localhost:3000.

## Environment variables

See [.env.local.example](.env.local.example). Key values:

- `MERCHANT_ADDRESS` / `NEXT_PUBLIC_MERCHANT_ADDRESS` — Devnet public key that must receive the payment.
- `SOLANA_RPC_URL` — defaults to `https://api.devnet.solana.com`.
- `MIN_LAMPORTS` — minimum payment (default `100000` = 0.0001 SOL).
- `REPOZIP_WORKSPACE` — repo the MCP server indexes (defaults to `process.cwd()`).
- `REPOZIP_DEV_UNLOCK` — set to `1` to bypass the paywall for local demos.
- `REPOZIP_LLM_ENRICH` — set to `1` to enable optional LLM enrichment of low-confidence file responsibilities (requires `OPENAI_API_KEY`). Off by default; deterministic heuristics always run.

## Payment flow (Devnet)

1. Connect a browser wallet.
2. Pay >= 0.0001 SOL to the merchant address.
3. The signature is verified server-side (recipient, amount, confirmation).
4. A verified signature unlocks the MCP payload once (replay-protected).

## Incremental updates

The index is persisted once and refreshed per file. On an edit, `updateFile` reparses only the changed file, rewrites its shard, and re-derives the (cheap) cross-file layers — no full repository rescan or reparse of unchanged files. A stale/corrupt or version-mismatched manifest triggers a one-time full rebuild.

## Security notes

Verification is server-side only; client-reported success is never trusted. The scanner refuses to read outside the workspace root and skips `node_modules`, VCS internals, `.env*`, and binaries. LLM enrichment (when enabled) sends only the compressed file descriptor, never raw source. Only `.env.local.example` (placeholders) is committed.

## License

MIT