MCP Documentation Scraper Template

A complete Python template for building Model Context Protocol (MCP) servers with documentation scraping, vector search, and flexible deployment options.

Features

• 🔍 Dual Search Modes: Semantic search (ML-powered) or keyword search (lightweight)

• 🤖 Multi-Provider Embeddings: HuggingFace (local), OpenAI, or Azure OpenAI

• 📚 Web Scraping: Configurable documentation crawler with depth control

• 💾 Vector Storage: ChromaDB with persistent storage

• ⏰ Auto-Updates: Scheduled documentation refresh

• 🔌 Dual Transport: stdio (local MCP) or HTTP/SSE (remote access)

• 🐳 Docker Ready: Containerized deployment with profiles

• ☸️ Kubernetes Ready: Production deployment with autoscaling

Related MCP server: MCP Docs Server

Quick Start

Option 1: stdio Mode (Local MCP - Default)

For: VS Code, Claude Desktop integration on your local machine

1. Configure environment:

   cp .env.example .env
   # Edit .env and set your DOCS_URLS
   # Keep TRANSPORT=stdio (or leave it out, stdio is default)

2. Start container:

   docker-compose up -d

   This starts mcp-server-template container (no ports exposed).

3. Configure MCP client (VS Code or Claude Desktop):

   {
     "mcpServers": {
       "docs-scraper": {
         "command": "docker",
         "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
       }
     }
   }

Option 2: HTTP Mode (Remote Access)

For: Testing API, remote access, or preparing for Kubernetes deployment

1. Configure environment for HTTP:

   cp .env.example .env
   # Edit .env and set:
   # TRANSPORT=http
   # DOCS_URLS=https://your-docs-site.com/

2. Start container with HTTP profile:

   docker-compose --profile http up -d

   This starts mcp-server-template-http container on port 3003.

3. Test the server:

   # Check health
   curl http://localhost:3003/health
   
   # MCP clients connect to:
   # http://localhost:3003/sse

Switching Between Modes

Stop current mode:

docker-compose down

Start stdio mode:

docker-compose up -d

Start HTTP mode:

docker-compose --profile http up -d

View logs:

# stdio mode
docker logs mcp-server-template -f

# HTTP mode
docker logs mcp-server-template-http -f

Configuration

Key environment variables (see .env.example for full list):

• TRANSPORT: Transport mode - stdio (local MCP) or http (remote/Kubernetes)

• DOCS_URLS: Documentation URLs to scrape (comma-separated for multiple sites)

  • Single: DOCS_URLS=https://docs.example.com/

  • Multiple: DOCS_URLS=https://docs.example.com/,https://api.example.com/docs/,https://guides.example.com/

• SWAGGER_URLS: Swagger/OpenAPI JSON URLs (comma-separated, optional)

• USE_EMBEDDINGS: Enable semantic search (true) or keyword search (false)

• EMBEDDING_PROVIDER: huggingface, openai, or azure

• CRAWL_MAX_DEPTH: Maximum crawl depth (0-3, recommended 2)

• AUTO_UPDATE_ENABLED: Enable scheduled documentation updates (true/false)

• HTTP_PORT: Port for HTTP mode (default: 3000, mapped to 3003 on host)

Transport Modes

stdio Mode (Local MCP)

Best for: Local development, VS Code, Claude Desktop

This server uses stdio transport for direct MCP client communication via stdin/stdout pipes.

VS Code (settings.json):

{
  "mcp.servers": {
    "docs-scraper": {
      "command": "docker",
      "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
    }
  }
}

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "docs-scraper": {
      "command": "docker",
      "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
    }
  }
}

HTTP Mode (Remote Access)

Best for: Remote access, multi-user, Kubernetes deployments

Set TRANSPORT=http in .env and use the HTTP profile:

docker-compose --profile http up -d

Endpoints:

• GET /health - Health check

• GET /sse - MCP Server-Sent Events endpoint

MCP Client Configuration (for remote HTTP clients):

{
  "mcpServers": {
    "docs-scraper": {
      "url": "http://localhost:3003/sse",
      "transport": "sse"
    }
  }
}

Kubernetes Deployment

See k8s-deployment.yaml for production Kubernetes deployment with:

• Horizontal Pod Autoscaling

• Persistent Volume Claims for ChromaDB

• Ingress configuration

• Health checks and readiness probes

Customization

This is a template - fork and customize for your documentation sources:

1. Update DOCS_URLS in .env

2. Customize scraping in src/mcp_server_template/documentation_scraper.py

3. Add custom tools in src/mcp_server_template/server.py

4. Adjust chunking and search parameters as needed

Requirements

• Python 3.12+

• Docker & Docker Compose (recommended)

• 500MB+ memory for semantic search (50MB for keyword-only)