# MCP Server Knowledge Engine
A powerful Model Context Protocol (MCP) server that transforms any PDF document collection into an intelligent, searchable knowledge base accessible through Claude Desktop. This server features advanced search capabilities using TF-IDF scoring, proximity matching, and domain-specific optimization.
## 🌟 Key Features
- **🔍 Advanced Search Engine**: TF-IDF-based inverted index with proximity matching for highly relevant results
- **📄 Universal PDF Support**: Process any PDF collection - technical docs, legal papers, research, and more
- **⚡ High Performance**: Cached search index, incremental processing, and background initialization
- **🎯 Domain Optimization**: Configure domain-specific keywords for enhanced search accuracy
- **⚙️ Fully Configurable**: JSON-based configuration with environment variable support
- **🛠️ Comprehensive CLI**: Complete server management through intuitive commands
- **🔗 Seamless MCP Integration**: Ready-to-use with Claude Desktop, VS Code, and other MCP clients
- **📊 Smart Caching**: MD5 hash-based change detection for efficient updates
## 📋 Quick Start
### Prerequisites
- Python 3.8 or higher
- pip (Python package manager)
- Claude Desktop app (for MCP integration)
### 1. Installation
```bash
# Clone the repository
git clone https://github.com/lhstorm/mcp_server_knowledge_engine.git
cd mcp_server_knowledge_engine
# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
```
### 2. Create Your Server
```bash
# Interactive setup
python manage_server.py create-config
# This will ask you for:
# - Server name (e.g., 'legal-docs-server')
# - Display name (e.g., 'Legal Documents Server')
# - PDF folder location
# - Domain-specific keywords
```
### 3. Add PDF Documents
```bash
# Add individual PDFs
python manage_server.py add-pdf /path/to/document.pdf
python manage_server.py add-pdf /path/to/another-doc.pdf
# Or copy PDFs directly to your configured folder
```
### 4. Process Documents
```bash
# Convert PDFs to searchable format
python manage_server.py process-pdfs
```
### 5. Generate MCP Configuration
```bash
# Generate configuration for Claude Desktop
python generate_mcp_config.py --merge
# Or get the config to copy manually
python generate_mcp_config.py
```
### 6. Start Using with Claude
Restart Claude Desktop and your server will appear in the MCP tools menu!
## 💬 Using with Claude Desktop
Once configured, you can interact with your PDFs naturally:
**Example prompts:**
- "Search for information about [topic] in the documentation"
- "What does the documentation say about [specific feature]?"
- "Find all references to [keyword] across all PDFs"
- "Show me the content of [document name]"
- "List all available documents"
**Advanced usage:**
- "Search for [term1] near [term2]" - Leverages proximity matching
- "Get page 15 of [document]" - Retrieves specific pages
- "Find the top 10 results for [query]" - Adjusts result count
## 📁 Project Structure
```
mcp_server_knowledge_engine/
├── server.py # Main MCP server with search engine
├── config.py # Configuration management & validation
├── manage_server.py # CLI for server management
├── generate_mcp_config.py # MCP configuration generator
├── convert_pdfs.py # Standalone PDF conversion utility
├── server_config.json # Active server configuration
├── requirements.txt # Python dependencies
├── examples/ # Example configurations
│ ├── legal_docs_config.json
│ ├── medical_docs_config.json
│ ├── research_papers_config.json
│ └── tech_docs_config.json
└── your-pdfs/ # Your PDF folder (configurable)
├── document1.pdf
├── document2.pdf
└── markdown/ # Auto-generated cache
├── .pdf_cache.json # Processing metadata
├── .search_index.pkl # Cached search index
├── document1.md # Converted documents
└── document2.md
```
## ⚙️ Configuration
The server is configured via `server_config.json`:
```json
{
"server": {
"name": "my-docs-server",
"display_name": "My Documents Server",
"description": "Search through my PDF collection",
"version": "1.0.0"
},
"storage": {
"pdf_folder": "./docs",
"markdown_folder": "./docs/markdown",
"domain_keywords": ["keyword1", "keyword2", "domain-term"]
},
"tools": {
"search": {
"name": "search_docs",
"description": "Search through PDF documentation"
},
"list": {
"name": "list_docs",
"description": "List all available documents"
},
"content": {
"name": "get_document_content",
"description": "Get full content from documents"
},
"max_results_default": 5
},
"processing": {
"cache_enabled": true,
"parallel_processing": true,
"max_file_size_mb": 50,
"context_size": 500
}
}
```
## 🛠️ Management Commands
### Server Management
```bash
# Create new configuration
python manage_server.py create-config
# Test configuration
python manage_server.py test
# Generate MCP config
python manage_server.py generate-mcp-config
```
### PDF Management
```bash
# List all PDFs
python manage_server.py list-pdfs
# Add PDF
python manage_server.py add-pdf document.pdf
# Remove PDF
python manage_server.py remove-pdf document.pdf
# Process all PDFs
python manage_server.py process-pdfs
```
### MCP Configuration
```bash
# Print MCP config
python generate_mcp_config.py
# Automatically merge with Claude Desktop config
python generate_mcp_config.py --merge
# Save to file
python generate_mcp_config.py --output my_mcp_config.json
```
## 💡 Usage Examples
### Legal Documents Server
```json
{
"server": {
"name": "legal-docs-server",
"display_name": "Legal Documents Server"
},
"storage": {
"domain_keywords": ["contract", "liability", "jurisdiction", "plaintiff", "defendant"]
}
}
```
### Technical Documentation Server
```json
{
"server": {
"name": "tech-docs-server",
"display_name": "Technical Documentation Server"
},
"storage": {
"domain_keywords": ["API", "function", "class", "method", "parameter", "return"]
}
}
```
### Research Papers Server
```json
{
"server": {
"name": "research-server",
"display_name": "Research Papers Server"
},
"storage": {
"domain_keywords": ["hypothesis", "methodology", "results", "conclusion", "analysis"]
}
}
```
## 🔧 Available MCP Tools
Each server provides three configurable tools:
1. **Search Tool** (default: `search_docs`)
- Intelligent search through all documents
- TF-IDF scoring with proximity matching
- Returns relevant excerpts with context
2. **List Tool** (default: `list_docs`)
- Lists all available documents
- Shows document metadata and page counts
3. **Content Tool** (default: `get_document_content`)
- Retrieves full document content
- Can fetch specific pages
- Includes complete markdown formatting
## 🎯 Domain Customization
The server adapts to your domain through:
- **Domain Keywords**: Configure terms important to your field
- **Tool Names**: Customize tool names (e.g., `search_legal_docs`)
- **Descriptions**: Tailor descriptions for your use case
- **Context Size**: Adjust how much context to return in search results
## 🔍 How the Search Engine Works
### Inverted Index Architecture
The server uses an advanced inverted index for lightning-fast searches:
1. **Document Processing**: PDFs are converted to markdown and tokenized
2. **Index Building**: Words are mapped to their locations (document, page, position)
3. **TF-IDF Scoring**:
- **TF (Term Frequency)**: How often a word appears in a document
- **IDF (Inverse Document Frequency)**: How rare a word is across all documents
- Combined score ensures relevant, unique results rank higher
### Search Features
- **Proximity Boosting**: Multi-word queries score higher when terms appear close together
- **Context Extraction**: Returns relevant snippets with search terms highlighted
- **Domain Keyword Recognition**: Configured keywords get special treatment
- **Page-Level Precision**: Results include specific page numbers
- **Smart Caching**: Search index persists between server restarts
## 📊 Performance Optimizations
- **Incremental Processing**: MD5 hash-based change detection - only new/modified PDFs are processed
- **Persistent Search Index**: Pickled index loads instantly on server restart
- **Background Initialization**: Server accepts connections while building index
- **Memory Efficiency**: Streaming PDF processing and markdown storage
- **Configurable Limits**: Control file size limits and processing parameters
## 🐛 Troubleshooting
### Common Issues & Solutions
**Server not appearing in Claude Desktop:**
- Ensure MCP configuration was merged: `python generate_mcp_config.py --merge`
- Check Python path: `which python` or `where python` (Windows)
- Verify server_config.json exists and is valid JSON
- Restart Claude Desktop after configuration changes
**PDFs not processing:**
- Check folder permissions: `ls -la /path/to/pdf/folder`
- Verify PDF files aren't corrupted: `file document.pdf`
- Look for errors in stderr: `python server.py 2>error.log`
- Ensure sufficient disk space for markdown cache
**Search returns no/poor results:**
- Initial indexing may take time - check stderr for progress
- Verify markdown files exist: `ls markdown/*.md`
- Check search index exists: `ls markdown/.search_index.pkl`
- Try single-word queries first, then expand
- Review domain keywords in configuration
**Server crashes or hangs:**
- Check Python version (3.8+ required): `python --version`
- Verify all dependencies installed: `pip install -r requirements.txt`
- Clear cache and reprocess: `rm -rf markdown/.pdf_cache.json markdown/.search_index.pkl`
- Check for file locking issues on Windows
### Debug Mode
```bash
# Run with full debug output
python server.py 2>&1 | tee debug.log
# Check server initialization
grep "initialization" debug.log
# Monitor PDF processing
grep "Processing\|Error" debug.log
```
### Validation Commands
```bash
# Test configuration validity
python manage_server.py test
# Verify configuration loading
python -c "from config import load_config_from_env_or_file; c=load_config_from_env_or_file(); print(f'✓ Config loaded: {c.server.name}')"
# Check MCP integration
python generate_mcp_config.py # Should output valid JSON
```
## 🚀 Advanced Usage
### Multiple Servers
You can run multiple specialized servers:
```bash
# Legal documents server
python manage_server.py --config legal_config.json create-config
# Technical docs server
python manage_server.py --config tech_config.json create-config
# Research papers server
python manage_server.py --config research_config.json create-config
```
### Batch Processing
```bash
# Process multiple PDF folders
for folder in docs legal_docs tech_docs; do
python convert_pdfs.py "$folder" "$folder/markdown"
done
```
### Custom Keywords
Configure domain-specific keywords for better search relevance:
```json
{
"storage": {
"domain_keywords": [
"algorithm", "data structure", "complexity",
"optimization", "performance", "scalability"
]
}
}
```
## 🏗️ Architecture Overview
### Core Components
1. **SearchIndex Class** (`server.py:27-140`)
- Implements inverted index with TF-IDF scoring
- Handles word tokenization and document indexing
- Provides proximity-based ranking for multi-word queries
2. **GenericPDFServer Class** (`server.py:142-661`)
- Main server implementation with MCP protocol handling
- Manages PDF processing pipeline
- Handles async operations and background initialization
3. **Configuration System** (`config.py`)
- Dataclass-based type-safe configuration
- JSON schema validation
- Environment variable support
4. **Management CLI** (`manage_server.py`)
- Interactive configuration creation
- PDF management operations
- Server testing and validation
### Data Flow
```
PDFs → PDF Reader → Markdown Converter → Search Index → MCP Tools → Claude
↓ ↓ ↓
[.pdf files] [.md cache files] [.search_index.pkl]
```
## 🔄 Current Server Configuration
The repository currently includes a configuration for QuantConnect documentation (`server_config.json`). To create your own server:
```bash
# Option 1: Interactive setup
python manage_server.py create-config
# Option 2: Copy and modify an example
cp examples/tech_docs_config.json server_config.json
# Edit server_config.json with your settings
```
## 📚 Example Use Cases
- **Legal Firms**: Search through contracts, case files, and legal documents
- **Research Labs**: Query scientific papers and technical reports
- **Software Teams**: Access API documentation and technical specs
- **Medical Practices**: Search patient records and medical literature
- **Educational Institutions**: Browse course materials and textbooks
## 🤝 Contributing
We welcome contributions! Here are some ways to help:
### Enhancement Ideas
1. **Document Format Support**: Add support for Word, HTML, or other formats
2. **Search Improvements**: Implement semantic search, fuzzy matching, or ML-based ranking
3. **Performance**: Add database backend, parallel processing, or distributed indexing
4. **Tools**: Create specialized MCP tools for specific domains
5. **UI**: Build a web interface for configuration management
### Development Guidelines
- Follow existing code style and patterns
- Add tests for new functionality
- Update documentation for new features
- Submit PRs with clear descriptions
## 🔐 Security Considerations
- The server only has read access to specified PDF folders
- No external network calls are made during operation
- Sensitive data remains local - nothing is sent to external services
- Configure appropriate file permissions for your PDF folders
## 📄 License
This project is open source. See LICENSE file for details.
## 🙏 Acknowledgments
Built with the [Model Context Protocol](https://modelcontextprotocol.io/) by Anthropic.
---
**Ready to transform your PDFs into a searchable knowledge base?**
Run `python manage_server.py create-config` to get started! 🚀
## 📦 Dependencies
- **mcp**: Model Context Protocol SDK for building MCP servers
- **PyPDF2**: PDF parsing and text extraction
- **asyncio**: Asynchronous I/O for concurrent operations
- **jsonschema**: JSON validation for configuration files
All dependencies are lightweight and have minimal system requirements.
