# nl-file-search

Natural-language search over local files. Phase 1 indexes markdown, plain text, images, videos, and PDFs with **Gemini Embedding 2**, stores vectors in SQLite (`sqlite-vec`), and exposes search through a CLI and a Cursor MCP server.

**Work in progress.** Designed by Gary Lucero. Coded by Cursor.

A later Python app can import the same `nl_file_search.search` module. Office documents are Phase 2; music is Phase 3. See [background/ROADMAP.md](background/ROADMAP.md).

## Phase 1 file types

| Type | Extensions | How it is indexed |
| --- | --- | --- |
| Text | `.md`, `.txt` | Heading/paragraph chunks |
| Images | `.png`, `.jpg`, `.jpeg`; `.webp`, `.bmp`, `.gif` converted to JPEG | Sent to Gemini as image bytes |
| Video | `.mp4`, `.mov` | Split into 120s clips with ffmpeg |
| PDF | `.pdf` | One page per embedding |

Unknown extensions are skipped. HEIC is skipped. Secret-like names (`.env`, `*.pem`, `credentials.json`, SSH keys) are never read or sent to Gemini.

## Requirements

- Windows, macOS, or Linux
- Python 3.12+
- A Gemini API key (`GEMINI_API_KEY`)
- [ffmpeg](https://ffmpeg.org/) on `PATH` when you index videos (Windows: `winget install Gyan.FFmpeg`; macOS: `brew install ffmpeg`; Linux: your package manager)
- Network access for every ingest and every search (queries are embedded with the same model)

SQLite is not Windows-only. The same code uses `Path.home() / "nl-file-search"` on every OS.

## Setup

Windows (PowerShell):

```powershell
cd C:\source\repos\nl-file-search
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -U pip
python -m pip install -e .
```

macOS / Linux:

```bash
cd ~/src/nl-file-search
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e .
```

Copy the example config and env files into `~/nl-file-search` (your home directory on every OS). Run this from the repo, with the venv active. `nl-search` does not create these files.

```bash
python scripts/copy_profile.py
```

Then:

1. Edit `~/nl-file-search/config.yaml` and set the folders to index.
2. Edit `~/nl-file-search/.env` and set `GEMINI_API_KEY=...` (never commit this file).

If a key was ever pasted into a chat or ticket, revoke it in Google AI Studio and issue a new one.

Data lives **outside the repo**:

| | Path |
| --- | --- |
| Data folder | `~/nl-file-search/` |
| SQLite index | `~/nl-file-search/index.sqlite` |

```
~/nl-file-search/
  config.yaml           # copied from config.example.yaml; you edit this
  .env                  # copied from .env.example; you put the API key here
  index.sqlite          # created on the first successful nl-search ingest
  logs/                 # created when a command runs after config exists
```

### Paths in `config.yaml`

List each folder under `sources` with a `path` value. Use **forward slashes** in double quotes; that form works on Windows, macOS, and Linux:

```yaml
sources:
  - path: "~/Notes"
  - path: "~/Pictures"
  - path: "/absolute/path/to/folder"
```

`~` is expanded to your home directory. Paths are resolved to absolute locations when ingest runs.

On Windows, a backslash in a **double-quoted** string is a YAML escape (`\U` in `C:\Users` is not a folder separator). Use one of these instead:

| Form | Example |
| --- | --- |
| Forward slashes | `"C:/Users/you/Notes"` |
| Single quotes | `'C:\Users\you\Notes'` |
| Escaped backslashes | `"C:\\Users\\you\\Notes"` |

Example `config.yaml` (also in [config.example.yaml](config.example.yaml)):

```yaml
sources:
  - path: "~/Notes"
  - path: "~/Pictures"
exclude:
  - "**/.git/**"
  - "**/node_modules/**"
embed:
  model: gemini-embedding-2
  dimensions: 768
video:
  max_seconds: 120
```

## CLI

```bash
nl-search ingest
nl-search ingest --path /path/to/folder
nl-search search "vintage red truck in the rain"
nl-search status
```

## Cursor MCP

Add a server in Cursor’s MCP settings (user or project). Point `command` at this repo’s venv Python:

```json
{
  "mcpServers": {
    "nl-file-search": {
      "command": "/absolute/path/to/nl-file-search/.venv/bin/python",
      "args": ["-m", "nl_file_search.mcp_server"]
    }
  }
}
```

On Windows, use `.venv\\Scripts\\python.exe` instead of `.venv/bin/python`. Restart Cursor MCP after saving. Do not put `GEMINI_API_KEY` in that config; the server reads `~/nl-file-search/.env`.

Tools:

- `search_files` — natural-language search; use this first when asking about indexed local files
- `get_file` — indexed text or media metadata for a path already in the index (not an arbitrary disk read)

## Security

- The API key lives only in `~/nl-file-search/.env`.
- The SQLite database lives only in `~/nl-file-search/index.sqlite`.
- Ingest skips credential-like files and default junk directories (`.git`, `node_modules`, `.venv`, `__pycache__`).
- `get_file` only returns rows already in the index. It will not open `..\..\.env` or other paths that were never ingested.
- Retrieved snippets go to Cursor the same way an open file would. Do not index folders that must never leave the machine.

## Phase 2 and 3 (not built yet)

- **Phase 2:** modern Office (`.docx`, `.xlsx`, `.pptx`)
- **Phase 3:** music (`.mp3`, `.m4a`, `.flac`)

Out of scope: Google Docs, Drive export, LibreOffice, legacy `.doc` / `.xls` / `.ppt`.