# Computer Index

A personal file-indexing system for `/home/Declan/`. It builds a SQLite/libSQL
database that catalogs every file with filesystem metadata, extracts text
content, chunks it, and generates vector embeddings for semantic search.

The pipeline runs in phases. Every script is standalone, re-runnable, and reads
its settings from `config.toml`.

```
scour  →  extract  →  chunk  →  dedup  →  embed  →  fts
(metadata) (text)   (splits) (dupes) (vectors) (keyword idx)
```

An MCP server (`serve.py`) exposes the finished index — semantic search, keyword
search, and file lookup — to agents and the MCP Inspector.

## Requirements

- **Python 3.14**, managed by [uv](https://docs.astral.sh/uv/)
- **ollama** with the embedding model pulled (Phase 2, Step 3 only)
- **NVIDIA GPU** (RTX 4060, 8 GB VRAM) for embeddings and PDF OCR

## Setup

```bash
uv sync                              # create the venv and install dependencies
ollama pull qwen3-embedding:4b       # embedding model (~2.5 GB), for embed.py
```

## Configuration

All paths and exclusions live in `config.toml`:

- `root_path` — directory to index (default `/home/Declan`)
- `db_path` — database file (default `computer_index.db`)
- `exclude_dirs.names` / `exclude_dirs.paths` — directories the scour skips

`schema.sql` is the DDL reference for the four core tables (`scour_runs`,
`files`, `file_contents`, `text_chunks`) plus the `file_fts` keyword index.

---

## Phase 1 — Scour (filesystem metadata)

`scour.py` walks `root_path` with `os.scandir`, skips excluded/inaccessible
directories, and records one metadata row per file into `files`. No file
contents are read here.

```bash
uv run scour.py                          # scan config root_path into computer_index.db
uv run scour.py --root /some/project     # scan a different directory (testing)
uv run scour.py --db /tmp/test.db        # write to a throwaway db (testing)
```

Prints a validation summary (counts by extension, depth, largest files).

### Report (optional)

`report.py` generates a self-contained `report.html` dashboard (Plotly) from the
latest scour run — treemap of disk usage, extension/depth/age breakdowns,
largest files.

```bash
uv run report.py                         # writes report.html
uv run report.py --db /tmp/test.db       # report on a different db
```

---

## Phase 2 — Text → Chunks → Embeddings

### Step 1 — Extract (`extract.py`)

Reads eligible files from the latest scour run and writes text into
`file_contents`. Three tiers:

- **Tier 1 — direct read:** code, document, and config files read as UTF-8.
  Data files are skipped as `data_file` (kept out of the search index): JSON
  always, and CSV/TSV/PSV when the content looks tabular.
- **Tier 2 — PDFs:** Unlimited-OCR to structured markdown (needs the separate
  GPU OCR stack: `uv sync --extra ocr`, ollama stopped so the OCR model fits VRAM).
- **Tier 3 — structured docs:** `.docx` (python-docx), `.ipynb` (nbformat).

```bash
uv run extract.py                        # all implemented tiers
uv run extract.py --tier 1               # direct-read files only
uv run extract.py --tier 3               # structured docs only
uv run extract.py --dry-run              # show what would be extracted
uv run extract.py --reextract            # re-extract files that already have a row
```

Prints extraction coverage by extension and files skipped by reason.

### Step 2 — Chunk (`chunk.py`)

Splits `file_contents` text into `text_chunks`. Prose/markdown is split
structurally (headers → paragraphs → sentences → words, ~4000-char target with
400-char overlap); code and config files are stored whole (truncated at 128K
chars).

```bash
uv run chunk.py                          # chunk everything not yet chunked
uv run chunk.py --stats                  # print chunk stats, no processing
uv run chunk.py --rechunk                # delete all chunks and redo
uv run chunk.py --file-id 12345          # chunk one file (testing)
```

Prints total chunks, per-file averages, token distribution, and sample chunks.

### Step 2.5 — Dedup (`dedup.py`)

Collapses identical-content files so only one copy is embedded and searched. For
each group of files sharing a `content_hash`, the shallowest path (min `depth`,
then lowest `file_id`) is kept and the others' chunks are deleted. Non-destructive
to disk, `files`, and `file_contents`; idempotent.

```bash
uv run dedup.py                          # remove duplicate-content chunks
uv run dedup.py --dry-run                # show what would be removed
```

`cleanup_data.py` is a related one-off: it applies the current data-file rule
(`extract.is_data_file`) to already-indexed files, deleting their chunks and
reclassifying them as `data_file`.

### Step 3 — Embed (`embed.py`)

Generates a `qwen3-embedding:4b` vector (2560 dims) for each chunk via ollama,
using libsql for the vector column. Requires the ollama server running:

```bash
ollama serve                             # in a separate terminal
```

```bash
uv run embed.py                          # embed all chunks with a NULL embedding
uv run embed.py --stats                  # embedding coverage
uv run embed.py --search "reinforcement learning policy gradient"
```

`embed.py` migrates the database to libsql and adds the `embedding` /
`embedding_model` columns on first run. Switching models (dimension change)
recreates the `embedding` column, so re-run `embed.py` to re-embed.

> The 4B embedder (~2.5 GB) and the Phase-2 PDF OCR model (~6 GB) do not both fit
> in 8 GB VRAM — stop ollama before running OCR.

### Step 4 — Keyword index (`fts.py`)

Builds `file_fts`, an external-content FTS5 mirror of `text_chunks.chunk_text`,
for the MCP server's keyword search. Indexing chunks (not whole files) keeps
keyword search aligned with semantic search — deduped and data-file content has
no chunks, so it stays out of both. Rebuilt in one pass (no triggers) — re-run
after any stage that changes `text_chunks` (chunk / dedup / cleanup).

```bash
uv run fts.py                            # (re)build the keyword index
uv run fts.py --status                   # indexed row count, no changes
```

---

## Phase 3 — MCP server (`serve.py`)

A [FastMCP](https://modelcontextprotocol.io) stdio server that exposes the
finished index to agents. Four read-only tools:

| tool | backend | use for |
|------|---------|---------|
| `search` | vector cosine (`embed.search`) | meaning-based / conceptual queries |
| `search_keyword` | FTS5 + BM25 (`file_fts`) | exact terms, names, literal phrases |
| `get_file_content` | `file_contents` / `text_chunks` | read a whole file, or a chunk window around a hit |
| `list_files` | `files` | browse by extension / name |

`search` and `search_keyword` return a `file_id` and a `chunk` index per hit;
pass both to `get_file_content(file_id, around_chunk=…)` to expand around a
specific chunk instead of dumping a whole document. Semantic queries are embedded
with the qwen3 query instruction (query-side only — no document re-embed).

```bash
uv add "mcp[cli]"                        # one-time: install the MCP SDK + Inspector
uv run mcp dev serve.py                  # interactive Inspector (browser UI)
uv run serve.py                          # run over stdio for an agent host
```

`search` needs the ollama server running (it embeds the query); the other tools
do not. Point an agent host at it with a stdio entry running `uv run serve.py`
from the project directory.

---

## Database schema

Four tables (`schema.sql`):

| table | populated by | holds |
|-------|--------------|-------|
| `scour_runs` | scour | one row per scan (root, timing, counts, status) |
| `files` | scour | filesystem metadata, one row per path |
| `file_contents` | extract | extracted text or skip reason, one row per file |
| `text_chunks` | chunk / embed | text chunks + embedding vectors |
| `file_fts` | fts | FTS5 keyword index over chunk text |

## Status

- **Phase 1 (scour):** complete
- **Phase 2:** extraction (Tier 1/2/3) · chunking · dedup · embeddings (4B) —
  complete
- **Phase 3 (Basic MCP server / agent access):** search / search_keyword /
  get_file_content / list_files over stdio — complete