# InsightFace-MCP

A **fully local, offline** facial recognition search system for your photo
library. It plugs into **LM Studio** via **MCP (Model Context Protocol)**, so
you can ask an LLM things like:

> "Find all pictures of this person in my Events folder."

...and get back every matching photo, with confidence scores, without a
single image or embedding ever leaving your machine.

```
LM Studio
   │  MCP tools
   ▼
Python MCP Server (server.py)
   ├── InsightFace   → detect faces, generate embeddings
   ├── FAISS         → store embeddings, fast similarity search
   └── Events/       → recursively scanned photo library
```

---

## 1. Project structure

```
InsightFace-MCP/
├── server.py          # MCP server (the entry point LM Studio launches)
├── face_engine.py      # shared helpers (model loading, I/O, FAISS, metadata)
├── index_faces.py      # scans Events/ and builds the face database
├── search_faces.py     # searches the database for a reference face
├── cluster_faces.py    # optional: group faces without a reference photo
├── config.py            # all settings live here
├── requirements.txt
├── README.md
├── database/
│   ├── faces.index       # FAISS vector index (created on first run)
│   ├── metadata.json      # per-face metadata (created on first run)
│   └── file_cache.json    # mtimes, for incremental re-indexing
├── known_faces/          # (optional) put reference photos here
└── Events/               # <- put your photo library here (or point config.py at it)
```

---

## 2. Installation (Windows)

### 2.1 Prerequisites

- **Python 3.10 or 3.11** (64-bit). Check with `python --version`.
- (Optional, for GPU speed) An **NVIDIA GPU** with a recent driver, plus a
  matching CUDA runtime. CPU-only works fine too, just slower on large
  libraries.

### 2.2 Create a virtual environment

Open **PowerShell** in the `InsightFace-MCP` folder:

```powershell
python -m venv venv
venv\Scripts\activate
```

### 2.3 Install dependencies

```powershell
pip install --upgrade pip
pip install -r requirements.txt
```

Notes:

- If you **do not** have an NVIDIA GPU, edit `requirements.txt` first:
  comment out `onnxruntime-gpu` and uncomment `onnxruntime`. Then re-run the
  install command. The server auto-detects whether CUDA is available and
  falls back to CPU either way, but installing the plain CPU package avoids
  downloading unused CUDA binaries.
- The first time InsightFace runs, it will automatically download the
  `buffalo_l` model pack (~300MB) from its own model repository (this is a
  one-time model download, not a cloud inference call — after this, all face
  detection/embedding happens locally with no further network access).
- `pillow-heif` gives you HEIC/HEIF (iPhone photo) support. If installation
  fails on your system, everything else still works — HEIC files will just be
  skipped with a warning.

### 2.4 Point the server at your photo library

Open `config.py` and set `EVENTS_FOLDER`, e.g.:

```python
EVENTS_FOLDER = Path(r"C:\Users\yourname\Pictures\Events")
```

Or set an environment variable instead of editing the file:

```powershell
setx EVENTS_FOLDER "C:\Users\yourname\Pictures\Events"
```

The folder is scanned **recursively** — every nested subfolder, no matter how
deep, is included. Supported formats: JPG, JPEG, PNG, WEBP, BMP, TIFF, and
HEIC/HEIF (if `pillow-heif` installed successfully).

### 2.5 Build the initial database

You can do this once up front (recommended for large libraries), or just let
LM Studio call `index_events_folder()` the first time you ask it to search.

```powershell
python index_faces.py
```

For 10,000+ photos, expect this to take a while on first run (minutes to
tens of minutes depending on CPU/GPU). Subsequent runs are incremental and
only process new/changed files.

### 2.6 (Optional) Add known reference photos

Drop a clear, front-facing photo of each person into `known_faces/`, e.g.
`known_faces/alice.jpg`. You can then just say "find Alice" in LM Studio and
point it at that file — see usage examples below.

---

## 3. LM Studio MCP setup

LM Studio supports MCP servers via its `mcp.json` configuration (Program
Files / Integrations, depending on your LM Studio version — check LM
Studio's own docs for the exact menu, since this changes between versions).

Add an entry like this, adjusting paths to match your machine:

```json
{
  "mcpServers": {
    "insightface-events-search": {
      "command": "C:\\path\\to\\InsightFace-MCP\\venv\\Scripts\\python.exe",
      "args": ["C:\\path\\to\\InsightFace-MCP\\server.py"],
      "env": {
        "EVENTS_FOLDER": "C:\\Users\\yourname\\Pictures\\Events"
      }
    }
  }
}
```

Key points:

- Use the **full path to the venv's `python.exe`**, not just `python`, so LM
  Studio uses the environment with `insightface`/`faiss` installed.
- `args` must point at the full path to `server.py`.
- The `env` block is optional if you already set `EVENTS_FOLDER` in
  `config.py` directly.

Restart LM Studio (or reload MCP servers) after editing the config. You
should see `insightface-events-search` listed as a connected MCP server with
four tools: `find_person`, `index_events_folder`, `database_status`, and
`list_clusters`.

---

## 4. Usage examples (in LM Studio chat)

Once connected, you can just talk naturally:

- **"Search my Events folder for this face"** — attach/reference a photo,
  and the model will call `find_person` with that image path.
- **"Show me every event where this person appears"** — same tool, phrased
  differently; the LLM decides to call `find_person`.
- **"Rebuild the face database"** — triggers `index_events_folder`.
- **"How many photos have you indexed so far?"** — triggers
  `database_status`.
- **"Who shows up most often in my photos?"** — triggers `list_clusters` to
  explore recurring faces without needing a reference photo first.

Note: exactly how you supply the reference image path depends on your LM
Studio version's file-attachment / tool-argument behavior. The simplest
reliable approach: save a reference photo into `known_faces/`, e.g.
`known_faces/mom.jpg`, then say:

> "Find every photo of the person in known_faces/mom.jpg"

The `find_person` tool will resolve that relative path automatically.

---

## 5. How it works

1. **Indexing (`index_faces.py`)**
   - Recursively walks `EVENTS_FOLDER` with `os.walk`, which by design never
     skips subdirectories.
   - Loads each supported image (via Pillow, so HEIC/WEBP/TIFF all work
     consistently), converts to the BGR array format InsightFace/OpenCV
     expect.
   - Runs InsightFace's `buffalo_l` detector, which returns bounding boxes,
     detection confidence, and a 512-dimension ArcFace embedding per face.
   - Faces below `MIN_DETECTION_SCORE` (config.py) are discarded as
     low-confidence/noise.
   - Each embedding is L2-normalized, then added to a FAISS
     `IndexFlatIP` (inner product on normalized vectors = cosine similarity)
     wrapped in `IndexIDMap2` so each vector has a stable integer ID.
   - Metadata per face (source file, filename, face index within that image,
     EXIF timestamp if present, bounding box, detection score) is stored in
     `metadata.json`, keyed by that same integer ID.
   - File modification times are cached in `file_cache.json`; on subsequent
     runs, unchanged files are skipped entirely (incremental indexing), and
     changed files have their old faces removed and re-added.

2. **Searching (`search_faces.py`)**
   - Loads your reference photo, detects the (largest, most prominent) face
     in it, and computes its normalized embedding the same way.
   - Queries the FAISS index for the most similar stored embeddings.
   - Groups results **by source image** (since one photo can contain
     multiple faces, and the same person might appear more than once in a
     group photo), keeping the best-matching face's score as that image's
     confidence.
   - Filters by `threshold` and returns results sorted by confidence,
     descending.

3. **Serving (`server.py`)**
   - Wraps the above in an MCP server using `FastMCP`, exposing
     `find_person`, `index_events_folder`, `database_status`, and
     `list_clusters` as callable tools with clear docstrings so the LLM in
     LM Studio understands when and how to call each one.
   - Runs over stdio, which is what LM Studio expects for locally-launched
     MCP servers.
   - No network calls are made at query time. The only network access ever
     used is the one-time InsightFace model download on first run.

---

## 6. Performance notes

- **FAISS `IndexFlatIP`** does an exact (not approximate) similarity search.
  For libraries up to roughly 100k-200k faces this is still fast (a few
  milliseconds to tens of milliseconds per query) since it's just a matrix
  multiply. If your library grows far beyond that, consider switching the
  index type in `face_engine.new_empty_index()` to an approximate index like
  `IndexIVFFlat` or `IndexHNSWFlat` for sub-linear search time.
- **GPU vs CPU**: `face_engine.get_face_app()` automatically tries CUDA via
  `onnxruntime`'s `CUDAExecutionProvider` first, and transparently falls back
  to CPU if CUDA isn't available or fails to initialize. No configuration
  needed — check the server's stderr log line on startup to confirm which
  provider was used.
- **Incremental indexing**: re-running `index_events_folder()` after adding
  a handful of new photos to a 10,000-photo library only processes the new
  files, not the whole set.
- **Batching**: detection is done one image at a time (InsightFace's
  `FaceAnalysis.get()` is per-image), which is the standard approach for
  variable-resolution photo libraries and keeps memory use predictable even
  on large folders.

---

## 7. Troubleshooting

**"No module named 'insightface'" or similar import errors**
Make sure you activated the venv (`venv\Scripts\activate`) before running
anything, and that LM Studio's `mcp.json` points at the venv's `python.exe`,
not your system Python.

**Server doesn't show up in LM Studio**
- Double-check `mcp.json` paths use double backslashes (`\\`) or forward
  slashes, and are absolute paths.
- Try running `python server.py` manually in your terminal first — if it
  crashes, LM Studio will fail silently. Fix any errors shown there first.

**"CUDAExecutionProvider" not found / falls back to CPU unexpectedly**
- Confirm `onnxruntime-gpu` installed successfully (`pip show onnxruntime-gpu`).
- Confirm your NVIDIA driver + CUDA toolkit versions are compatible with the
  installed `onnxruntime-gpu` version (check onnxruntime's release notes).
- This isn't fatal — CPU fallback still works, just slower.

**HEIC files are being skipped**
- Confirm `pillow-heif` installed: `pip show pillow-heif`.
- Some very new/exotic HEIC variants (e.g. certain burst-mode formats) may
  still fail to decode; check the server log for the specific error.

**`database_status` shows `database_healthy: false`**
- This means the FAISS index and metadata.json have drifted out of sync
  (e.g. a crash mid-write). Run `python index_faces.py --full` to rebuild
  cleanly from scratch.

**Searches return no matches / too many false positives**
- Too few matches: lower `threshold` (e.g. from 0.45 to 0.35) when calling
  `find_person`, or use a clearer, well-lit, front-facing reference photo.
- Too many false positives: raise `threshold` (e.g. to 0.55-0.6).
- Faces at extreme angles, heavy occlusion (masks, sunglasses), or very low
  resolution are inherently harder to match — this is a property of any
  face-recognition model, not specific to this setup.

**Indexing is very slow**
- Confirm you're actually using the GPU if you have one (check startup log).
- For very large libraries, consider running `index_faces.py` directly from
  the terminal once (rather than through LM Studio) so you can watch the
  `tqdm` progress bar and confirm throughput.

**"No face detected in reference image"**
- Use a photo where the face is reasonably large, front-facing, and not
  heavily obscured. Cropping the reference photo tighter around the face
  can help.

---

## 8. Privacy & security

- Everything — model inference, embedding generation, similarity search —
  runs on your machine.
- No images, embeddings, or metadata are uploaded anywhere.
- The only outbound network request in the entire system is InsightFace's
  one-time model weight download the first time it runs (standard for any
  local ML library); after that, the server works fully offline.