README.md•16.9 kB
# AnyDocs MCP Server
[](https://python.org)
[](https://github.com/modelcontextprotocol/python-sdk)
[](LICENSE)
[](https://github.com/psf/black)
> Transform any website's documentation section into an MCP-compatible server using the Python MCP SDK.
## 🚀 Overview
AnyDocs MCP Server is a comprehensive solution that turns **any website's documentation** into an interactive, AI-accessible knowledge base through the Model Context Protocol (MCP). It can scrape, index, and serve documentation from any website - from modern API docs to legacy documentation portals.
### Key Features
- **🌐 Universal Website Scraping**: Turn ANY website's documentation into an interactive knowledge base
- **🔌 Universal Adapter System**: Support for GitBook, Notion, Confluence, and custom documentation platforms
- **🔍 Advanced Search**: Full-text search with SQLite FTS and semantic search capabilities
- **🔐 Robust Authentication**: API Key, OAuth2, and JWT-based authentication
- **⚡ High Performance**: Async/await architecture with caching and rate limiting
- **🎛️ Web Management Interface**: FastAPI-based admin panel for configuration and monitoring
- **📊 Real-time Monitoring**: Health checks, metrics, and logging
- **🐳 Docker Ready**: Complete containerization with development and production configurations
- **🔄 Auto-sync**: Automatic content synchronization with source documentation
## 📋 Requirements
- Python 3.11+ (recommended: 3.11 or 3.12)
- SQLite 3.35+ (for FTS5 support)
- Optional: Redis (for caching)
- Optional: PostgreSQL/MySQL (for production)
## 🛠️ Installation
### Using uvx (Recommended)
The easiest way to run AnyDocs MCP Server is using `uvx`, which automatically manages dependencies and virtual environments:
```bash
# Run directly with uvx (no installation needed)
uvx anydocs-mcp-server
# Run with custom configuration
uvx anydocs-mcp-server --config config.yaml
# Run in debug mode
uvx anydocs-mcp-server --debug
# Install globally with uvx for repeated use
uvx install anydocs-mcp-server
# Then run anytime with:
anydocs-mcp-server --config config.yaml
```
### Quick Start
```bash
# Clone the repository
git clone https://github.com/funky1688/anydocs-mcp.git
cd anydocs-mcp
# Install dependencies using uv (recommended)
pip install uv
uv pip install -e .
# Copy environment configuration
cp .env.example .env
# Copy and customize configuration
cp config.yaml my-config.yaml
# Edit configuration (add your API keys and settings)
nano .env
# Start the server (hybrid mode - MCP + Web interface)
uv run python start.py
```
### Manual Installation
```bash
# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
uv pip install -e . # for production
# OR for development:
uv pip install -e .[dev]
# Copy and configure environment
cp .env.example .env
# Edit configuration files
nano .env
# Initialize and start
uv run python start.py --mode hybrid --debug
```
### Running as a Python Module
After installation, you can also run the server as a Python module:
```bash
# Run as module
python -m anydocs_mcp
# With configuration
python -m anydocs_mcp --config config.yaml
# Debug mode
python -m anydocs_mcp --debug
```
### Docker Installation
```bash
# Development environment
docker-compose -f docker-compose.dev.yml up -d
# Production environment
docker-compose up -d
```
## 🚀 Usage
### Starting the Server
AnyDocs MCP Server supports 3 startup modes:
#### 1. Hybrid Mode (Default - Recommended)
Starts both MCP server and web management interface simultaneously:
```bash
uv run python start.py
# or explicitly:
uv run python start.py --mode hybrid
```
- **MCP Server**: Available at `http://localhost:8000` (handles MCP protocol communication)
- **Web Interface**: Available at `http://localhost:8080` for management
- **Best for**: Most users who want both MCP functionality and web management
> **Important**: Always use `uv run` to ensure the correct virtual environment is used.
#### 2. MCP Server Only
Starts only the MCP server without web interface:
```bash
uv run python start.py --mode mcp
```
- **Use case**: Production deployments where only MCP protocol is needed
- **Lighter resource usage**: No web interface overhead
- **Best for**: Headless servers, CI/CD environments
#### 3. Web Interface Only
Starts only the web management interface:
```bash
uv run python start.py --mode web
```
- **Use case**: Administrative tasks, configuration management
- **Web Interface**: Available at `http://localhost:8080`
- **Best for**: Configuration, monitoring, and testing without MCP protocol
#### Additional Options
```bash
# Debug mode with auto-reload
uv run python start.py --debug
# Custom configuration file
uv run python start.py --config custom-config.yaml
# Skip dependency check (faster startup)
uv run python start.py --no-deps-check
# Kill occupied ports before starting
uv run python start.py --kill-ports
# Skip database initialization
uv run python start.py --no-db-init
```
### Command Line Options
```bash
uv run python start.py --help
# Available options:
# --mode {mcp,web,hybrid} Startup mode (default: hybrid)
# --config CONFIG Configuration file path (default: config.yaml)
# --debug Enable debug mode
# --no-deps-check Skip dependency check
# --no-db-init Skip database initialization
# --kill-ports Kill occupied ports before starting
```
### Web Management Interface
Access the web interface at `http://localhost:8080` to:
- Configure document sources
- Manage users and API keys
- Monitor system health
- View logs and metrics
- Test MCP endpoints
### MCP Client Integration
```python
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
# Connect to AnyDocs MCP Server
async with stdio_client(StdioServerParameters(
command="python",
args=["main.py"]
)) as (read, write):
async with ClientSession(read, write) as session:
# Initialize the connection
await session.initialize()
# List available tools
tools = await session.list_tools()
# Search documents
result = await session.call_tool(
"search_documents",
arguments={"query": "authentication", "limit": 10}
)
```
## 📚 Documentation Adapters
### Supported Platforms
| Platform | Status | Features |
|----------|--------|----------|
| **Any Website** | ✅ | **Universal scraper for any documentation site** |
| GitBook | ✅ | Full API integration, real-time sync |
| Notion | ✅ | Database and page content, webhooks |
| Confluence | ✅ | Space and page management, attachments |
| GitHub | ✅ | Repository documentation, wikis |
| GitLab | ✅ | Project documentation, wikis |
| SharePoint | ✅ | Document libraries, lists |
| Slack | ✅ | Channel messages, knowledge base |
| File System | ✅ | Local markdown files, watch mode |
| Custom | 🔧 | Extensible adapter framework |
### Adding a New Adapter
```python
from anydocs_mcp.adapters.base import BaseDocumentAdapter
class CustomAdapter(BaseDocumentAdapter):
"""Custom documentation adapter implementation."""
async def fetch_documents(self) -> List[Document]:
"""Fetch documents from your platform."""
# Implementation here
pass
async def get_document_content(self, doc_id: str) -> str:
"""Get specific document content."""
# Implementation here
pass
```
## 🔧 Configuration
### Troubleshooting
#### Common Installation Issues
**Python-Jose Import Error**: If you encounter `No module named 'jose'` error:
```bash
# Always use uv run to ensure correct virtual environment
uv run python start.py
# If the issue persists, reinstall python-jose with cryptography extras
uv pip uninstall python-jose
uv pip install "python-jose[cryptography]"
```
**Dependency Check Failures**: If you see errors about missing `pyyaml` or `beautifulsoup4`:
```bash
# The dependency check has been fixed to use correct import names
# Ensure you're using the latest version
git pull origin main
uv pip install -e .
```
**Configuration Attribute Errors**: If you see `'AppConfig' object has no attribute 'server_host'`:
```bash
# This is fixed in the current version - ensure you have the latest code
git pull origin main
```
**Virtual Environment Issues**: If packages seem installed but imports fail:
```bash
# Always use 'uv run' to ensure correct environment
uv run python start.py
# Check if you're in the right environment
which python # Should point to your project's Python
uv pip list # Should show installed packages
```
**Setup.py Conflicts**: If you encounter conflicts with multiple setup files:
```bash
# The redundant root setup.py has been removed
# Use pyproject.toml for package management
uv pip install -e .
```
#### Environment Setup Best Practices
1. **Always use `uv run`** for executing Python scripts to ensure correct environment
2. **Use `uv pip install`** instead of `uv install` for package installation
3. **Check virtual environment** with `uv pip list` if imports fail
4. **Pull latest changes** if you encounter configuration issues
#### Port Conflicts
If ports 8000 or 8080 are occupied:
```bash
# Kill processes on required ports (Windows)
netstat -ano | findstr :8000
taskkill /F /PID <PID>
# Or use the built-in option
uv run python start.py --kill-ports
```
### Environment Variables
```bash
# Server Configuration
ANYDOCS_HOST=localhost
ANYDOCS_PORT=8000
ANYDOCS_WEB_PORT=8080
ANYDOCS_DEBUG=false
# Database
DATABASE_URL=sqlite:///data/anydocs.db
# DATABASE_URL=postgresql://user:pass@localhost/anydocs
# Authentication
JWT_SECRET_KEY=your-secret-key
API_KEY_PREFIX=anydocs_
# Document Adapters
GITBOOK_API_TOKEN=your-gitbook-token
NOTION_API_TOKEN=your-notion-token
CONFLUENCE_API_TOKEN=your-confluence-token
# Cache (Optional)
REDIS_URL=redis://localhost:6379/0
# Monitoring
ENABLE_METRICS=true
LOG_LEVEL=INFO
```
### YAML Configuration
```yaml
# config.yaml
server:
host: localhost
port: 8000
web_port: 8080
debug: false
database:
url: sqlite:///data/anydocs.db
pool_size: 10
echo: false
auth:
jwt_secret: ${JWT_SECRET_KEY}
token_expire_minutes: 1440
api_key:
prefix: anydocs_
length: 32
adapters:
gitbook:
api_token: ${GITBOOK_API_TOKEN}
base_url: https://api.gitbook.com
rate_limit: 100
notion:
api_token: ${NOTION_API_TOKEN}
version: "2022-06-28"
rate_limit: 3
```
## 🔍 MCP Tools
AnyDocs MCP Server provides the following tools:
### Core Tools
- **search_documents** - Search documents with full-text and semantic search
- **get_document** - Retrieve a specific document by ID
- **list_sources** - List all configured document sources
- **summarize_content** - Summarize document content
- **ask_question** - Ask questions about document content
### AI-Powered Tools
- **generate_documentation** - AI-assisted documentation generation
- **translate_content** - Multi-language content translation
- **extract_insights** - Extract insights and analytics from documentation
- **suggest_improvements** - AI-powered content enhancement suggestions
## 🧪 Development
### Setup Development Environment
```bash
# Install development dependencies
uv pip install -e .[dev]
# Setup pre-commit hooks
uv run pre-commit install
# Run tests
uv run pytest
# Run with coverage
uv run pytest --cov=src/anydocs_mcp
# Code formatting
uv run black src/ tests/
uv run isort src/ tests/
# Type checking
uv run mypy src/
# Security checks
uv run bandit -r src/
```
### Project Structure
```
anydocs-mcp/
├── src/anydocs_mcp/ # Main package
│ ├── adapters/ # Document adapters
│ ├── auth/ # Authentication
│ ├── config/ # Configuration management
│ ├── content/ # Content processing
│ ├── database/ # Database models and operations
│ ├── utils/ # Utilities and helpers
│ ├── web/ # Web interface
│ └── server.py # MCP server implementation
├── tests/ # Test suite
├── docs/ # Documentation
├── scripts/ # Utility scripts
│ └── setup.py # Development environment setup
├── pyproject.toml # Package configuration (modern Python packaging)
├── start.py # Main startup script
└── main.py # MCP server entry point
```
> **Note**: The project uses `pyproject.toml` for package configuration following modern Python packaging standards. The redundant root `setup.py` has been removed to avoid conflicts.
### Running Tests
```bash
# All tests
uv run pytest
# Unit tests only
uv run pytest tests/unit/
# Integration tests only
uv run pytest tests/integration/
# With coverage
uv run pytest --cov=src/anydocs_mcp --cov-report=html
# Performance tests
uv run pytest tests/performance/
```
## 📊 Monitoring & Observability
### Health Checks
```bash
# Check service health
curl http://localhost:8080/health
# Detailed health check
curl http://localhost:8080/health/detailed
```
### Metrics
Metrics are available at `/metrics` endpoint in Prometheus format:
- Request count and duration
- Database connection pool status
- Document sync statistics
- Error rates and types
- Cache hit/miss ratios
### Logging
Structured logging with configurable levels:
```python
# Application logs
tail -f anydocs_mcp.log
# Check logs in real-time
uv run python start.py --debug
```
## 🐳 Docker Deployment
### Development
```bash
# Start development environment
docker-compose -f docker-compose.dev.yml up -d
# View logs
docker-compose logs -f anydocs-mcp-dev
# Access shell
docker-compose exec anydocs-mcp-dev bash
```
### Production
```bash
# Build and start production environment
docker-compose up -d
# Scale services
docker-compose up -d --scale anydocs-mcp=3
# Update services
docker-compose pull && docker-compose up -d
```
### Monitoring Stack
```bash
# Start with monitoring
docker-compose --profile monitoring up -d
# Access services
# Grafana: http://localhost:3001 (admin/admin)
# Prometheus: http://localhost:9090
```
## 🔒 Security
### Authentication Methods
1. **API Keys**: Simple token-based authentication
2. **JWT Tokens**: Stateless authentication with expiration
3. **OAuth2**: Integration with external providers
### Security Best Practices
- All API endpoints require authentication
- Rate limiting on all endpoints
- Input validation and sanitization
- SQL injection prevention
- CORS configuration
- Security headers
- Audit logging
### Security Scanning
```bash
# Run security checks
uv run bandit -r src/
# Dependency vulnerability scan
uv run safety check
# SAST scanning
uv run bandit -r src/
```
## 🚀 Performance
### Optimization Features
- **Async/Await**: Non-blocking I/O operations
- **Connection Pooling**: Efficient database connections
- **Caching**: Redis-based caching with TTL
- **Rate Limiting**: Prevent API abuse
- **Batch Processing**: Efficient bulk operations
- **Lazy Loading**: On-demand content loading
### Performance Monitoring
```bash
# Performance testing
uv run pytest tests/performance/
# Memory profiling
uv run python -m memory_profiler start.py
```
## 🤝 Contributing
We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
### Development Workflow
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Run the test suite
6. Submit a pull request
### Code Standards
- Follow PEP 8 style guide
- Use type hints
- Write comprehensive tests
- Document public APIs
- Use meaningful commit messages
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## 🙏 Acknowledgments
- [Model Context Protocol](https://github.com/modelcontextprotocol) for the MCP specification
- [Python MCP SDK](https://github.com/modelcontextprotocol/python-sdk) for the SDK implementation
- All contributors and maintainers
## 📞 Support
- 📧 Email: team@anydocs-mcp.com
- 🐛 Issues: [GitHub Issues](https://github.com/your-org/anydocs-mcp/issues)
- 📖 Documentation: [Full Documentation](https://docs.anydocs-mcp.com)
---
**Made with ❤️ by funky1688**