# web-docs-mcp

**MCP server for searching web documentation via sitemap.xml**

Index and search multiple web documentation sites using [Model Context Protocol (MCP)](https://modelcontextprotocol.io). Works with Claude Code, Cursor, Windsurf, Continue.dev, and Zed.

## Features

- 🗺️ **Sitemap-based indexing** - Automatically discover pages via sitemap.xml
- 🔒 **Basic authentication support** - Access password-protected documentation sites
- 🔍 **Vector search** - Semantic search powered by Ollama embeddings
- 📊 **Incremental sync** - Only re-scrape changed pages based on lastmod dates
- 🎯 **Smart content extraction** - Automatically identifies main content area
- 🚀 **Fast setup** - Auto-configure Claude Code with one command

## Quick Start

```bash
# Install globally
npm install -g @vdpeijl/web-docs-mcp

# Run interactive setup
web-docs-mcp init

# Sync documentation
web-docs-mcp sync

# Configure your MCP client
web-docs-mcp setup claude-code
```

## How It Works

```
Sitemap.xml → Web Scraper → Content Extraction → Text Chunking → Ollama Embeddings → SQLite + sqlite-vec
                                                                                              ↓
MCP Clients ←→ MCP Server (stdio) ←→ Vector Search ←→ Ranked Results
```

1. **Discovery**: Fetches URLs from sitemap.xml (supports sitemap indexes)
2. **Scraping**: Downloads HTML pages (with optional basic auth)
3. **Extraction**: Identifies main content using common selectors (`<main>`, `<article>`, etc.)
4. **Chunking**: Splits text into ~500 token chunks with 50 token overlap
5. **Embedding**: Generates vector embeddings via Ollama (`nomic-embed-text`)
6. **Indexing**: Stores in SQLite with vector search (sqlite-vec)
7. **Search**: Semantic search across all indexed documentation

## Requirements

- **Node.js** 20+
- **Ollama** with `nomic-embed-text` model

Install Ollama from [ollama.com](https://ollama.com), then:

```bash
ollama pull nomic-embed-text
```

## Configuration

Configuration is stored in `~/.config/web-docs-mcp/config.json`:

```json
{
  "ollama": {
    "baseUrl": "http://localhost:11434",
    "model": "nomic-embed-text"
  },
  "sync": {
    "chunkSize": 500,
    "chunkOverlap": 50
  },
  "requests": {
    "delayMs": 100,
    "timeout": 30000,
    "userAgent": "web-docs-mcp/1.0"
  },
  "sources": [
    {
      "id": "myco-docs",
      "name": "My Company Docs",
      "baseUrl": "https://docs.myco.com",
      "sitemapUrl": "https://docs.myco.com/sitemap.xml",
      "enabled": true
    }
  ]
}
```

### Adding Sources with Authentication

For password-protected documentation:

```bash
# Add source interactively
web-docs-mcp sources add

# Or via command line
web-docs-mcp sources add \
  --id myco-internal \
  --name "Internal Docs" \
  --url https://internal.myco.com \
  --sitemapUrl https://internal.myco.com/sitemap.xml
```

Set credentials via environment variables:

```bash
export DOCS_USERNAME="your-username"
export DOCS_PASSWORD="your-password"
```

Or create a `.env` file in your working directory:

```bash
DOCS_USER=your-username
DOCS_PASSWORD=your-password
```

### Automated Setup with .env

You can configure your entire source in a `.env` file for automated setup:

```bash
# Source configuration
DOCS_SOURCE_ID=myco-docs
DOCS_SOURCE_NAME=My Company Docs
DOCS_BASE_URL=https://docs.myco.com
DOCS_SITEMAP_URL=https://docs.myco.com/sitemap.xml

# Authentication (optional)
DOCS_USER=your-username
DOCS_PASSWORD=your-password
```

When running `web-docs-mcp init`, it will automatically detect and use these values if present.

Configure in `config.json`:

```json
{
  "auth": {
    "usernameEnvVar": "DOCS_USERNAME",
    "passwordEnvVar": "DOCS_PASSWORD"
  }
}
```

## CLI Commands

```bash
# Interactive setup wizard
web-docs-mcp init

# Sync all enabled sources
web-docs-mcp sync

# Manage sources
web-docs-mcp sources list
web-docs-mcp sources add
web-docs-mcp sources remove <id>
web-docs-mcp sources enable <id>
web-docs-mcp sources disable <id>

# Auto-configure MCP client
web-docs-mcp setup claude-code

# View statistics
web-docs-mcp stats

# Run diagnostics
web-docs-mcp doctor

# Start MCP server (for manual setup)
web-docs-mcp serve

# Uninstall
web-docs-mcp uninstall              # Remove data and configs
web-docs-mcp uninstall --keep-data  # Remove configs only
```

## MCP Tools

### `search_knowledge_base`

Search web documentation with semantic similarity.

**Parameters:**
- `query` (string, required) - Natural language search query
- `sources` (array, optional) - Filter by source IDs
- `limit` (number, optional) - Max results (default: 5, max: 20)

**Example:**
```
Search: "How do I configure authentication?"
```

### `list_sources`

List all configured documentation sources and sync status.

**Returns:** Source name, URL, sitemap, page count, chunk count, last synced date

## Incremental Sync

The sync process is optimized for efficiency:

1. **Sitemap Fetch**: Downloads and parses sitemap.xml
2. **Change Detection**: Compares `<lastmod>` dates with database
3. **Selective Scraping**: Only fetches new or updated pages
4. **Cleanup**: Removes pages no longer in sitemap

If a page has no `<lastmod>` in the sitemap, it will be re-scraped on every sync (safer default when change detection isn't possible).

## Smart Content Extraction

The scraper attempts to identify the main content area using these selectors (in order):

1. `<main>`
2. `<article>`
3. `[role="main"]`
4. `.content`, `.main-content`, `#content`
5. `.documentation`, `.doc-content`

Fallback: Removes `<nav>`, `<header>`, `<footer>`, `<aside>` and processes remaining content.

## Data Storage

- **Config**: `~/.config/web-docs-mcp/config.json`
- **Database**: `~/.local/share/web-docs-mcp/kb.sqlite`
- **Logs**: `~/.local/share/web-docs-mcp/logs/`

## MCP Client Setup

### Claude Code (Automatic)

```bash
web-docs-mcp setup claude-code
```

This runs:
```bash
claude mcp add --transport stdio web-docs -- bash -l -c "web-docs-mcp serve"
```

The shell wrapper (`bash -l -c`) ensures compatibility with node version managers (fnm, nvm, volta, asdf).

### Manual Setup

Add to your MCP client config:

```json
{
  "mcpServers": {
    "web-docs": {
      "command": "bash",
      "args": ["-l", "-c", "web-docs-mcp serve"]
    }
  }
}
```

## Debugging

Enable debug logs:

```bash
DEBUG=web-docs-mcp:* web-docs-mcp sync
```

## Advanced Usage

### Multiple Documentation Sources

Index different sites independently:

```bash
web-docs-mcp sources add --id react --name "React Docs" \
  --url https://react.dev --sitemapUrl https://react.dev/sitemap.xml

web-docs-mcp sources add --id vue --name "Vue Docs" \
  --url https://vuejs.org --sitemapUrl https://vuejs.org/sitemap.xml
```

Search specific sources:

```
Search React docs: How do I use hooks?
(MCP tool will filter by source: ["react"])
```

### Custom Chunk Size

Adjust for your embedding model:

```json
{
  "sync": {
    "chunkSize": 750,
    "chunkOverlap": 100
  }
}
```

### Rate Limiting

Control request delays:

```json
{
  "requests": {
    "delayMs": 500,  // 500ms between requests
    "timeout": 60000  // 60 second timeout
  }
}
```

## Limitations

- Requires sites to have sitemap.xml
- Basic authentication only (no OAuth/SSO)
- Single-threaded scraping (respects rate limits)
- JavaScript-rendered content not supported (static HTML only)

## Troubleshooting

### "Ollama not found"
```bash
# Install Ollama from ollama.com, then:
ollama serve
ollama pull nomic-embed-text
```

### "Authentication failed"
- Verify environment variables are set correctly
- Check credentials work in browser
- Ensure basic auth is used (not OAuth/SSO)

### "No results found"
- Run `web-docs-mcp stats` to verify pages are indexed
- Check if sync completed successfully
- Try more specific search queries

### "Sitemap not found"
- Verify sitemap URL is correct (usually `/sitemap.xml`)
- Check if site uses sitemap index (`sitemap_index.xml`)
- Some sites use `/robots.txt` to specify sitemap location

## License

MIT

## Contributing

Issues and PRs welcome at [https://github.com/vdpeijl/web-docs-mcp](https://github.com/vdpeijl/web-docs-mcp)