MCP Document Indexer

A Python-based MCP (Model Context Protocol) server for local document indexing and search using LanceDB vector database and local LLMs.

Features

• Real-time Document Monitoring: Automatically indexes new and modified documents in configured folders

• Multi-format Support: Handles PDF, Word (docx/doc), text, Markdown, and RTF files

• Local LLM Integration: Uses Ollama for document summarization and keyword extraction. Nothing ever leaves your computer

• Vector Search: Semantic search using LanceDB and sentence transformers

• MCP Integration: Exposes search and catalog tools via Model Context Protocol

• Incremental Indexing: Only processes changed files to save resources

• Performance Optimized: Designed for decent performance on standard laptops (e.g. M1/M2 MacBook)

Installation

Prerequisites

1. Python 3.9+ installed

2. uv package manager:

3. Ollama (for local LLM):

Install MCP Document Indexer

Configuration

Configure the indexer using environment variables or a .env file:

Usage

Run as Standalone Service

Integrate with Claude Desktop

Add to your Claude Desktop configuration (~/Library/Application Support/Claude/claude_desktop_config.json):

MCP Tools

The indexer exposes the following tools via MCP:

search_documents

Search for documents using natural language queries.

• Parameters:

  • query: Search query text

  • limit: Maximum number of results (default: 10)

  • search_type: "documents" or "chunks"

get_catalog

List all indexed documents with summaries.

• Parameters:

  • skip: Number of documents to skip (default: 0)

  • limit: Maximum documents to return (default: 100)

get_document_info

Get detailed information about a specific document.

• Parameters:

  • file_path: Path to the document

reindex_document

Force reindexing of a specific document.

• Parameters:

  • file_path: Path to the document to reindex

get_indexing_stats

Get current indexing statistics.

Example Usage in Claude

Once configured, you can use the indexer in Claude:

Architecture

File Processing Pipeline

1. File Detection: Watchdog monitors configured folders for changes

2. Document Parsing: Extracts text from PDF, Word, and text files

3. Text Chunking: Splits documents into overlapping chunks for better retrieval

4. LLM Processing: Generates summaries and extracts keywords using Ollama

5. Embedding Generation: Creates vector embeddings using sentence transformers

6. Vector Storage: Stores documents and chunks in LanceDB

7. MCP Exposure: Makes search and catalog tools available via MCP

Performance Considerations

• Incremental Indexing: Only changed files are reprocessed

• Async Processing: Parallel processing of multiple documents

• Batch Operations: Efficient batch indexing for multiple files

• Debouncing: Prevents duplicate processing of rapidly changing files

• Size Limits: Configurable maximum file size to prevent memory issues

Troubleshooting

Ollama Not Available

If Ollama is not running or the model isn't available, the indexer falls back to simple text extraction without summarization.

Permission Issues

Ensure the indexer has read access to monitored folders:

Memory Usage

For large document collections, consider:

• Reducing CHUNK_SIZE to create smaller chunks

• Limiting MAX_FILE_SIZE_MB to skip very large files

• Using a smaller embedding model

Development

Running Tests

Code Formatting

Building Package

License

MIT License - See LICENSE file for details

Contributing

Contributions are welcome! Please:

1. Fork the repository

2. Create a feature branch

3. Add tests for new functionality

4. Submit a pull request

Support

For issues or questions:

• Open an issue on GitHub

• Check the troubleshooting section

• Review logs in the console output