# MCP Documentation Scraper Template

A complete Python template for building [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) servers with documentation scraping, vector search, and flexible deployment options.

## Features

- 🔍 **Dual Search Modes**: Semantic search (ML-powered) or keyword search (lightweight)
- 🤖 **Multi-Provider Embeddings**: HuggingFace (local), OpenAI, or Azure OpenAI
- 📚 **Web Scraping**: Configurable documentation crawler with depth control
- 💾 **Vector Storage**: ChromaDB with persistent storage
- ⏰ **Auto-Updates**: Scheduled documentation refresh
- 🔌 **Dual Transport**: stdio (local MCP) or HTTP/SSE (remote access)
- 🐳 **Docker Ready**: Containerized deployment with profiles
- ☸️ **Kubernetes Ready**: Production deployment with autoscaling

## Quick Start

### Option 1: stdio Mode (Local MCP - Default)

**For:** VS Code, Claude Desktop integration on your local machine

1. **Configure environment**:
   ```bash
   cp .env.example .env
   # Edit .env and set your DOCS_URLS
   # Keep TRANSPORT=stdio (or leave it out, stdio is default)
   ```

2. **Start container**:
   ```bash
   docker-compose up -d
   ```
   
   This starts `mcp-server-template` container (no ports exposed).

3. **Configure MCP client** (VS Code or Claude Desktop):
   ```json
   {
     "mcpServers": {
       "docs-scraper": {
         "command": "docker",
         "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
       }
     }
   }
   ```

### Option 2: HTTP Mode (Remote Access)

**For:** Testing API, remote access, or preparing for Kubernetes deployment

1. **Configure environment for HTTP**:
   ```bash
   cp .env.example .env
   # Edit .env and set:
   # TRANSPORT=http
   # DOCS_URLS=https://your-docs-site.com/
   ```

2. **Start container with HTTP profile**:
   ```bash
   docker-compose --profile http up -d
   ```
   
   This starts `mcp-server-template-http` container on port 3003.

3. **Test the server**:
   ```bash
   # Check health
   curl http://localhost:3003/health
   
   # MCP clients connect to:
   # http://localhost:3003/sse
   ```

### Switching Between Modes

**Stop current mode:**
```bash
docker-compose down
```

**Start stdio mode:**
```bash
docker-compose up -d
```

**Start HTTP mode:**
```bash
docker-compose --profile http up -d
```

**View logs:**
```bash
# stdio mode
docker logs mcp-server-template -f

# HTTP mode
docker logs mcp-server-template-http -f
```

## Configuration

Key environment variables (see `.env.example` for full list):

- `TRANSPORT`: Transport mode - `stdio` (local MCP) or `http` (remote/Kubernetes)
- `DOCS_URLS`: Documentation URLs to scrape (comma-separated for multiple sites)
  - Single: `DOCS_URLS=https://docs.example.com/`
  - Multiple: `DOCS_URLS=https://docs.example.com/,https://api.example.com/docs/,https://guides.example.com/`
- `SWAGGER_URLS`: Swagger/OpenAPI JSON URLs (comma-separated, optional)
- `USE_EMBEDDINGS`: Enable semantic search (`true`) or keyword search (`false`)
- `EMBEDDING_PROVIDER`: `huggingface`, `openai`, or `azure`
- `CRAWL_MAX_DEPTH`: Maximum crawl depth (0-3, recommended 2)
- `AUTO_UPDATE_ENABLED`: Enable scheduled documentation updates (`true`/`false`)
- `HTTP_PORT`: Port for HTTP mode (default: 3000, mapped to 3003 on host)

## Transport Modes

### stdio Mode (Local MCP)

**Best for:** Local development, VS Code, Claude Desktop

This server uses **stdio transport** for direct MCP client communication via stdin/stdout pipes.

**VS Code** (`settings.json`):
```json
{
  "mcp.servers": {
    "docs-scraper": {
      "command": "docker",
      "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
    }
  }
}
```

**Claude Desktop** (`claude_desktop_config.json`):
```json
{
  "mcpServers": {
    "docs-scraper": {
      "command": "docker",
      "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
    }
  }
}
```

### HTTP Mode (Remote Access)

**Best for:** Remote access, multi-user, Kubernetes deployments

Set `TRANSPORT=http` in `.env` and use the HTTP profile:

```bash
docker-compose --profile http up -d
```

**Endpoints:**
- `GET /health` - Health check
- `GET /sse` - MCP Server-Sent Events endpoint

**MCP Client Configuration** (for remote HTTP clients):
```json
{
  "mcpServers": {
    "docs-scraper": {
      "url": "http://localhost:3003/sse",
      "transport": "sse"
    }
  }
}
```

### Kubernetes Deployment

See `k8s-deployment.yaml` for production Kubernetes deployment with:
- Horizontal Pod Autoscaling
- Persistent Volume Claims for ChromaDB
- Ingress configuration
- Health checks and readiness probes

## Customization

This is a **template** - fork and customize for your documentation sources:

1. Update `DOCS_URLS` in `.env`
2. Customize scraping in `src/mcp_server_template/documentation_scraper.py`
3. Add custom tools in `src/mcp_server_template/server.py`
4. Adjust chunking and search parameters as needed

## Requirements

- Python 3.12+
- Docker & Docker Compose (recommended)
- 500MB+ memory for semantic search (50MB for keyword-only)