web-docs-mcp

MCP server for searching web documentation via sitemap.xml

Index and search multiple web documentation sites using Model Context Protocol (MCP). Works with Claude Code, Cursor, Windsurf, Continue.dev, and Zed.

Features

• 🗺️ Sitemap-based indexing - Automatically discover pages via sitemap.xml

• 🔒 Basic authentication support - Access password-protected documentation sites

• 🔍 Vector search - Semantic search powered by Ollama embeddings

• 📊 Incremental sync - Only re-scrape changed pages based on lastmod dates

• 🎯 Smart content extraction - Automatically identifies main content area

• 🚀 Fast setup - Auto-configure Claude Code with one command

Related MCP server: searxng-mcp

Quick Start

# Install globally
npm install -g @vdpeijl/web-docs-mcp

# Run interactive setup
web-docs-mcp init

# Sync documentation
web-docs-mcp sync

# Configure your MCP client
web-docs-mcp setup claude-code

How It Works

Sitemap.xml → Web Scraper → Content Extraction → Text Chunking → Ollama Embeddings → SQLite + sqlite-vec
                                                                                              ↓
MCP Clients ←→ MCP Server (stdio) ←→ Vector Search ←→ Ranked Results

1. Discovery: Fetches URLs from sitemap.xml (supports sitemap indexes)

2. Scraping: Downloads HTML pages (with optional basic auth)

3. Extraction: Identifies main content using common selectors (<main>, <article>, etc.)

4. Chunking: Splits text into ~500 token chunks with 50 token overlap

5. Embedding: Generates vector embeddings via Ollama (nomic-embed-text)

6. Indexing: Stores in SQLite with vector search (sqlite-vec)

7. Search: Semantic search across all indexed documentation

Requirements

• Node.js 20+

• Ollama with nomic-embed-text model

Install Ollama from ollama.com, then:

ollama pull nomic-embed-text

Configuration

Configuration is stored in ~/.config/web-docs-mcp/config.json:

{
  "ollama": {
    "baseUrl": "http://localhost:11434",
    "model": "nomic-embed-text"
  },
  "sync": {
    "chunkSize": 500,
    "chunkOverlap": 50
  },
  "requests": {
    "delayMs": 100,
    "timeout": 30000,
    "userAgent": "web-docs-mcp/1.0"
  },
  "sources": [
    {
      "id": "myco-docs",
      "name": "My Company Docs",
      "baseUrl": "https://docs.myco.com",
      "sitemapUrl": "https://docs.myco.com/sitemap.xml",
      "enabled": true
    }
  ]
}

Adding Sources with Authentication

For password-protected documentation:

# Add source interactively
web-docs-mcp sources add

# Or via command line
web-docs-mcp sources add \
  --id myco-internal \
  --name "Internal Docs" \
  --url https://internal.myco.com \
  --sitemapUrl https://internal.myco.com/sitemap.xml

Set credentials via environment variables:

export DOCS_USERNAME="your-username"
export DOCS_PASSWORD="your-password"

Or create a .env file in your working directory:

DOCS_USER=your-username
DOCS_PASSWORD=your-password

Automated Setup with .env

You can configure your entire source in a .env file for automated setup:

# Source configuration
DOCS_SOURCE_ID=myco-docs
DOCS_SOURCE_NAME=My Company Docs
DOCS_BASE_URL=https://docs.myco.com
DOCS_SITEMAP_URL=https://docs.myco.com/sitemap.xml

# Authentication (optional)
DOCS_USER=your-username
DOCS_PASSWORD=your-password

When running web-docs-mcp init, it will automatically detect and use these values if present.

Configure in config.json:

{
  "auth": {
    "usernameEnvVar": "DOCS_USERNAME",
    "passwordEnvVar": "DOCS_PASSWORD"
  }
}

CLI Commands

# Interactive setup wizard
web-docs-mcp init

# Sync all enabled sources
web-docs-mcp sync

# Manage sources
web-docs-mcp sources list
web-docs-mcp sources add
web-docs-mcp sources remove <id>
web-docs-mcp sources enable <id>
web-docs-mcp sources disable <id>

# Auto-configure MCP client
web-docs-mcp setup claude-code

# View statistics
web-docs-mcp stats

# Run diagnostics
web-docs-mcp doctor

# Start MCP server (for manual setup)
web-docs-mcp serve

# Uninstall
web-docs-mcp uninstall              # Remove data and configs
web-docs-mcp uninstall --keep-data  # Remove configs only

MCP Tools

search_knowledge_base

Search web documentation with semantic similarity.

Parameters:

• query (string, required) - Natural language search query

• sources (array, optional) - Filter by source IDs

• limit (number, optional) - Max results (default: 5, max: 20)

Example:

Search: "How do I configure authentication?"

list_sources

List all configured documentation sources and sync status.

Returns: Source name, URL, sitemap, page count, chunk count, last synced date

Incremental Sync

The sync process is optimized for efficiency:

1. Sitemap Fetch: Downloads and parses sitemap.xml

2. Change Detection: Compares <lastmod> dates with database

3. Selective Scraping: Only fetches new or updated pages

4. Cleanup: Removes pages no longer in sitemap

If a page has no <lastmod> in the sitemap, it will be re-scraped on every sync (safer default when change detection isn't possible).

Smart Content Extraction

The scraper attempts to identify the main content area using these selectors (in order):

1. <main>

2. <article>

3. [role="main"]

4. .content, .main-content, #content

5. .documentation, .doc-content

Fallback: Removes <nav>, <header>, <footer>, <aside> and processes remaining content.

Data Storage

• Config: ~/.config/web-docs-mcp/config.json

• Database: ~/.local/share/web-docs-mcp/kb.sqlite

• Logs: ~/.local/share/web-docs-mcp/logs/

MCP Client Setup

Claude Code (Automatic)

web-docs-mcp setup claude-code

This runs:

claude mcp add --transport stdio web-docs -- bash -l -c "web-docs-mcp serve"

The shell wrapper (bash -l -c) ensures compatibility with node version managers (fnm, nvm, volta, asdf).

Manual Setup

Add to your MCP client config:

{
  "mcpServers": {
    "web-docs": {
      "command": "bash",
      "args": ["-l", "-c", "web-docs-mcp serve"]
    }
  }
}

Debugging

Enable debug logs:

DEBUG=web-docs-mcp:* web-docs-mcp sync

Advanced Usage

Multiple Documentation Sources

Index different sites independently:

web-docs-mcp sources add --id react --name "React Docs" \
  --url https://react.dev --sitemapUrl https://react.dev/sitemap.xml

web-docs-mcp sources add --id vue --name "Vue Docs" \
  --url https://vuejs.org --sitemapUrl https://vuejs.org/sitemap.xml

Search specific sources:

Search React docs: How do I use hooks?
(MCP tool will filter by source: ["react"])

Custom Chunk Size

Adjust for your embedding model:

{
  "sync": {
    "chunkSize": 750,
    "chunkOverlap": 100
  }
}

Rate Limiting

Control request delays:

{
  "requests": {
    "delayMs": 500,  // 500ms between requests
    "timeout": 60000  // 60 second timeout
  }
}

Limitations

• Requires sites to have sitemap.xml

• Basic authentication only (no OAuth/SSO)

• Single-threaded scraping (respects rate limits)

• JavaScript-rendered content not supported (static HTML only)

Troubleshooting

"Ollama not found"

# Install Ollama from ollama.com, then:
ollama serve
ollama pull nomic-embed-text

"Authentication failed"

• Verify environment variables are set correctly

• Check credentials work in browser

• Ensure basic auth is used (not OAuth/SSO)

"No results found"

• Run web-docs-mcp stats to verify pages are indexed

• Check if sync completed successfully

• Try more specific search queries

"Sitemap not found"

• Verify sitemap URL is correct (usually /sitemap.xml)

• Check if site uses sitemap index (sitemap_index.xml)

• Some sites use /robots.txt to specify sitemap location

License

MIT

Contributing

Issues and PRs welcome at https://github.com/vdpeijl/web-docs-mcp