Computer Index

A personal file-indexing system for /home/Declan/. It builds a SQLite/libSQL database that catalogs every file with filesystem metadata, extracts text content, chunks it, and generates vector embeddings for semantic search.

The pipeline runs in phases. Every script is standalone, re-runnable, and reads its settings from config.toml.

scour  →  extract  →  chunk  →  dedup  →  embed  →  fts
(metadata) (text)   (splits) (dupes) (vectors) (keyword idx)

An MCP server (serve.py) exposes the finished index — semantic search, keyword search, and file lookup — to agents and the MCP Inspector.

Requirements

• Python 3.14, managed by uv

• ollama with the embedding model pulled (Phase 2, Step 3 only)

• NVIDIA GPU (RTX 4060, 8 GB VRAM) for embeddings and PDF OCR

Related MCP server: MemoryMesh

Setup

uv sync                              # create the venv and install dependencies
ollama pull qwen3-embedding:4b       # embedding model (~2.5 GB), for embed.py

Configuration

All paths and exclusions live in config.toml:

• root_path — directory to index (default /home/Declan)

• db_path — database file (default computer_index.db)

• exclude_dirs.names / exclude_dirs.paths — directories the scour skips

schema.sql is the DDL reference for the four core tables (scour_runs, files, file_contents, text_chunks) plus the file_fts keyword index.

Phase 1 — Scour (filesystem metadata)

scour.py walks root_path with os.scandir, skips excluded/inaccessible directories, and records one metadata row per file into files. No file contents are read here.

uv run scour.py                          # scan config root_path into computer_index.db
uv run scour.py --root /some/project     # scan a different directory (testing)
uv run scour.py --db /tmp/test.db        # write to a throwaway db (testing)

Prints a validation summary (counts by extension, depth, largest files).

Report (optional)

report.py generates a self-contained report.html dashboard (Plotly) from the latest scour run — treemap of disk usage, extension/depth/age breakdowns, largest files.

uv run report.py                         # writes report.html
uv run report.py --db /tmp/test.db       # report on a different db

Phase 2 — Text → Chunks → Embeddings

Step 1 — Extract (extract.py)

Reads eligible files from the latest scour run and writes text into file_contents. Three tiers:

• Tier 1 — direct read: code, document, and config files read as UTF-8. Data files are skipped as data_file (kept out of the search index): JSON always, and CSV/TSV/PSV when the content looks tabular.

• Tier 2 — PDFs: Unlimited-OCR to structured markdown (needs the separate GPU OCR stack: uv sync --extra ocr, ollama stopped so the OCR model fits VRAM).

• Tier 3 — structured docs: .docx (python-docx), .ipynb (nbformat).

uv run extract.py                        # all implemented tiers
uv run extract.py --tier 1               # direct-read files only
uv run extract.py --tier 3               # structured docs only
uv run extract.py --dry-run              # show what would be extracted
uv run extract.py --reextract            # re-extract files that already have a row

Prints extraction coverage by extension and files skipped by reason.

Step 2 — Chunk (chunk.py)

Splits file_contents text into text_chunks. Prose/markdown is split structurally (headers → paragraphs → sentences → words, ~4000-char target with 400-char overlap); code and config files are stored whole (truncated at 128K chars).

uv run chunk.py                          # chunk everything not yet chunked
uv run chunk.py --stats                  # print chunk stats, no processing
uv run chunk.py --rechunk                # delete all chunks and redo
uv run chunk.py --file-id 12345          # chunk one file (testing)

Prints total chunks, per-file averages, token distribution, and sample chunks.

Step 2.5 — Dedup (dedup.py)

Collapses identical-content files so only one copy is embedded and searched. For each group of files sharing a content_hash, the shallowest path (min depth, then lowest file_id) is kept and the others' chunks are deleted. Non-destructive to disk, files, and file_contents; idempotent.

uv run dedup.py                          # remove duplicate-content chunks
uv run dedup.py --dry-run                # show what would be removed

cleanup_data.py is a related one-off: it applies the current data-file rule (extract.is_data_file) to already-indexed files, deleting their chunks and reclassifying them as data_file.

Step 3 — Embed (embed.py)

Generates a qwen3-embedding:4b vector (2560 dims) for each chunk via ollama, using libsql for the vector column. Requires the ollama server running:

ollama serve                             # in a separate terminal

uv run embed.py                          # embed all chunks with a NULL embedding
uv run embed.py --stats                  # embedding coverage
uv run embed.py --search "reinforcement learning policy gradient"

embed.py migrates the database to libsql and adds the embedding / embedding_model columns on first run. Switching models (dimension change) recreates the embedding column, so re-run embed.py to re-embed.

The 4B embedder (~2.5 GB) and the Phase-2 PDF OCR model (~6 GB) do not both fit in 8 GB VRAM — stop ollama before running OCR.

Step 4 — Keyword index (fts.py)

Builds file_fts, an external-content FTS5 mirror of text_chunks.chunk_text, for the MCP server's keyword search. Indexing chunks (not whole files) keeps keyword search aligned with semantic search — deduped and data-file content has no chunks, so it stays out of both. Rebuilt in one pass (no triggers) — re-run after any stage that changes text_chunks (chunk / dedup / cleanup).

uv run fts.py                            # (re)build the keyword index
uv run fts.py --status                   # indexed row count, no changes

Phase 3 — MCP server (serve.py)

A FastMCP stdio server that exposes the finished index to agents. Four read-only tools:

search and search_keyword return a file_id and a chunk index per hit; pass both to get_file_content(file_id, around_chunk=…) to expand around a specific chunk instead of dumping a whole document. Semantic queries are embedded with the qwen3 query instruction (query-side only — no document re-embed).

uv add "mcp[cli]"                        # one-time: install the MCP SDK + Inspector
uv run mcp dev serve.py                  # interactive Inspector (browser UI)
uv run serve.py                          # run over stdio for an agent host

search needs the ollama server running (it embeds the query); the other tools do not. Point an agent host at it with a stdio entry running uv run serve.py from the project directory.

Database schema

Four tables (schema.sql):

Status

• Phase 1 (scour): complete

• Phase 2: extraction (Tier 1/2/3) · chunking · dedup · embeddings (4B) — complete

• Phase 3 (Basic MCP server / agent access): search / search_keyword / get_file_content / list_files over stdio — complete