Semantic Image Search MCP Server

Search your photo archive using natural language with AI-powered semantic understanding. Built as an MCP (Model Context Protocol) server for seamless integration with Claude Desktop.

Features

• Semantic Search: Find images by describing what's in them, not just filenames

• Zero Configuration: No manual tagging required - works out of the box

• EXIF Metadata: Automatically extracts camera settings, dates, and GPS data

• Fast Indexing: Optimized for Apple Silicon (MPS) and NVIDIA GPUs (CUDA)

• Claude Integration: Works natively with Claude Desktop via MCP

• Privacy First: Runs 100% locally - your photos never leave your machine

• Cloud-Synced Libraries: Index "online-only" files (OneDrive Files On-Demand, iCloud Drive, Dropbox) without keeping the whole library on disk - change detection reads placeholder metadata, so reindexing never re-downloads what it already knows

• Incremental & Scheduled Reindexing: Embeds only new or changed photos, with a scheduler agent to keep the index current automatically

Related MCP server: Claude RAG MCP Pipeline

Quick Start

1. Installation

# Clone the repository
git clone https://github.com/himalayantrust/semantic-image-search-mcp
cd semantic-image-search-mcp

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

2. Configuration

# Copy example configuration
cp config.yml.example config.yml

# Edit config.yml with your photo archive path
nano config.yml  # or use your preferred editor

Minimal configuration:

archive_path: "/path/to/your/photos"

3. Index Your Photos

# Run initial indexing
python3 -c "
import asyncio
from pathlib import Path
from src.config import Config
from src.indexer import ImageIndexer

async def index():
    config = Config.from_yaml(Path('config.yml'))
    indexer = ImageIndexer(config)
    stats = await indexer.index_archive()
    print(f'Indexed {stats[\"indexed\"]} images')

asyncio.run(index())
"

4. Set Up Claude Desktop Integration

Add to your Claude Desktop configuration (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "semantic-image-search": {
      "command": "python3",
      "args": ["/absolute/path/to/photo-library/run_server.py"],
      "env": {
        "PYTHONPATH": "/absolute/path/to/photo-library"
      }
    }
  }
}

5. Restart Claude Desktop

After updating the configuration, restart Claude Desktop. You should see the semantic-image-search server connected in the MCP section.

Usage Examples

Search for Images

Ask Claude:

Search my photos for images with people in classrooms

Find photos of mountain landscapes taken in 2024

Show me portraits with natural lighting

Get Image Details

Get detailed information about image abc123def456

View Archive Statistics

Show me statistics about my photo archive

Reindex After Adding Photos

Reindex my photo archive

Indexing Large or Cloud-Synced Libraries

If your archive lives in a cloud folder with "online-only" files (OneDrive Files On-Demand, iCloud Drive "Optimize Storage", Dropbox online-only), you can index the whole library without keeping it all on disk.

How online-only indexing works. The indexer detects new or changed files from each file's size and modification time, which it reads from the placeholder without downloading the file. Only images that are genuinely new or changed get materialised and embedded, so reindexing an unchanged library downloads nothing.

Folder-by-folder driver. For a large library on a storage-constrained machine, index_library.py indexes one allow-listed folder at a time so you can free space between folders:

# Index specific top-level folders (smallest first validates fast)
python3 index_library.py --config config.yml \
  --only "2019 Trip" --only "2020 Trip" --no-evict

# Or drive it from an allow-list file (one folder name per line)
cp folders.allow.example.txt folders.allow.txt   # then edit
python3 index_library.py --config config.yml --folders folders.allow.txt

• --only NAME (repeatable) or --folders FILE: which top-level folders to index

• --max-gb N: warn before indexing a folder larger than N GB (default 50)

• --no-evict: don't prompt to free space between folders (use for unattended runs)

After a folder is indexed, its thumbnails and embeddings are stored locally, so you can safely return the originals to online-only ("Free Up Space") and reclaim the disk. Only image files are ever read, so videos and other large files in the same tree are never downloaded.

Exact, training-free search index. The FAISS index uses IndexFlatL2 (exact nearest-neighbour) for libraries up to ~200k images. It needs no training step and searches tens of thousands of images in a few milliseconds.

Keeping the Index Current Automatically

reindex_missing.py embeds only new or changed images (using the size/mtime detection above) and rebuilds the search index:

python3 reindex_missing.py

To run it on a schedule, auto_reindex.sh wraps it with logging, and the bundled launchd agent runs it for you. com.himalayantrust.photo-reindex.plist is set to run weekly - edit its StartCalendarInterval for a different cadence:

cp com.himalayantrust.photo-reindex.plist ~/Library/LaunchAgents/
launchctl load -w ~/Library/LaunchAgents/com.himalayantrust.photo-reindex.plist

Incremental runs download and embed newly added photos and leave them local until you next free space. For a large new drop (tens of GB), use the attended index_library.py so eviction keeps peak disk in check.

MCP Tools

The server exposes four tools to Claude:

1. search_images

Search images using natural language queries with optional filters.

Parameters:

• query (string, required): Natural language description

• limit (integer, optional): Max results (default: 10, max: 100)

• date_from (string, optional): Filter by date (ISO format: YYYY-MM-DD)

• date_to (string, optional): Filter by date (ISO format: YYYY-MM-DD)

• folder_pattern (string, optional): Filter by folder path

Example:

{
  "query": "person standing in a room",
  "limit": 5,
  "date_from": "2024-01-01"
}

2. get_image_info

Get detailed metadata for a specific image.

Parameters:

• image_id (string, required): Unique image identifier

3. reindex_archive

Re-index the photo archive for new or modified images.

Parameters:

• force (boolean, optional): Force re-index all images (default: false)

4. get_archive_stats

Get statistics about the indexed photo archive.

No parameters required.

Configuration Reference

# Path to your photo archive (required)
archive_path: "/path/to/photos"

# Directory for storing index data (optional)
data_dir: "./data"

# CLIP model configuration
clip:
  # Model to use for embeddings
  model_name: "openai/clip-vit-base-patch32"  # or "openai/clip-vit-large-patch14"

  # Device for inference
  device: "auto"  # auto, mps, cuda, or cpu

  # Batch size for processing
  batch_size: 32  # Increase for more RAM/VRAM

# Search configuration
search:
  default_limit: 10
  max_limit: 100
  similarity_threshold: 0.0  # 0.0 = show all ranked results

# Thumbnail configuration
thumbnails:
  enabled: true
  max_size: 512
  quality: 85

Architecture

Technology Stack

• CLIP: OpenAI's vision-language model for understanding images

• FAISS: Facebook's vector similarity search library

• SQLite: Lightweight database for metadata storage

• MCP: Model Context Protocol for Claude integration

• PyTorch: ML framework with Apple Silicon (MPS) support

How It Works

1. Indexing:

   • Scans your archive for image files

   • Extracts EXIF metadata (camera, date, location, etc.)

   • Generates semantic embeddings using CLIP

   • Stores embeddings in FAISS vector index

   • Saves metadata in SQLite database

2. Searching:

   • Converts your text query to an embedding

   • Searches FAISS index for similar image embeddings

   • Applies filters (date, folder, etc.)

   • Returns ranked results with similarity scores

3. MCP Integration:

   • Exposes search tools to Claude via stdio protocol

   • Claude can search, get details, and manage your archive

   • All processing happens locally on your machine

Performance

Indexing Speed (Apple Silicon M-series)

• Small archives (< 1,000 images): ~30 seconds

• Medium archives (1,000 - 10,000 images): 2-5 minutes

• Large archives (10,000+ images): 10-30 minutes

Search Latency

• Typical query: 200-500ms

• With filters: 300-700ms

Memory Usage

• Base: ~200MB (model + server)

• Per 10,000 images: ~20MB (embeddings + metadata)

Troubleshooting

"FAISS index not found" Error

Run indexing first:

python3 -c "import asyncio; from src.indexer import ImageIndexer; from src.config import Config; from pathlib import Path; asyncio.run(ImageIndexer(Config.from_yaml(Path('config.yml'))).index_archive())"

MCP Server Not Connecting

1. Check Claude Desktop logs: ~/Library/Logs/Claude/mcp*.log

2. Verify absolute paths in claude_desktop_config.json

3. Ensure config.yml exists in the project directory

4. Check mcp-server.log for errors

Slow Indexing

• Reduce batch_size in config.yml (uses less memory, slightly slower)

• Check that MPS/CUDA is being used (look for "Using Apple Silicon MPS" message)

• Close other applications to free up RAM

Import Errors

Ensure virtual environment is activated:

source venv/bin/activate  # On Windows: venv\Scripts\activate

Development

Running Tests

pytest tests/

Code Formatting

black src/
ruff check src/

Use Cases

Museums & Archives

Search historical photo collections by content, era, or subject matter.

NGOs & Field Work

Find photos from specific trips, locations, or events for reports and social media.

Media Companies

Quickly locate stock footage and images matching creative briefs.

Photographers

Organize and search large portfolio collections by visual content.

Researchers

Find specific images in large datasets for analysis and publication.

Contributing

Contributions welcome! Please:

1. Fork the repository

2. Create a feature branch

3. Make your changes

4. Add tests if applicable

5. Submit a pull request

License

MIT License - see LICENSE file for details.

Acknowledgments

• Built on CLIP by OpenAI

• Uses FAISS by Meta AI Research

• Implements Model Context Protocol by Anthropic

Support

For issues and questions:

• GitHub Issues: https://github.com/himalayantrust/semantic-image-search-mcp/issues

• Email: info@himalayantrust.org

Built with love by the Himalayan Trust team 🏔️