# api-docs-mcp

[![Python](https://img.shields.io/badge/Python-3.10+-blue.svg)](https://www.python.org/)

[![MCP](https://img.shields.io/badge/MCP_Server-stdio-green.svg)](https://modelcontextprotocol.io/)

An MCP (Model Context Protocol) server that gives AI coding assistants access to up-to-date API documentation via Retrieval-Augmented Generation (RAG). Crawl any documentation site, index it into a vector store, and query it semantically — so your LLM always has accurate, current docs at its fingertips.

## Problem

LLMs have stale or incomplete knowledge of specific APIs and libraries. This project solves that by letting you:

1. **Crawl** any documentation website
2. **Index** the content into a local vector database
3. **Query** it semantically through an MCP server

AI assistants (like GitHub Copilot, Claude Desktop, Cursor, etc.) can then retrieve precise, current API documentation on demand.

## How It Works

```
┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Crawling   │ ──▶│   Chunking   │ ──▶│  Embedding   │ ──▶│   Storage    │
│  (crawl4ai)  │     │ (headings +  │     │ (sentence-   │     │  (ChromaDB)  │
│              │     │  paragraphs) │     │ transformers)│     │              │
└──────────────┘     └──────────────┘     └──────────────┘     └──────────────┘
                                                                   ▲
┌──────────────┐     ┌──────────────┐                              │
│   MCP Client │ ──▶│  Retrieval   │ ──▶  Similarity Search ─────┘
│  (Copilot,   │     │   Engine     │
│   Claude,    │     │              │
│   Cursor…)   │     └──────────────┘
└──────────────┘
```

### Pipeline Stages

1. **Crawling** — Uses `crawl4ai` (headless browser) to recursively crawl a documentation site from a root URL, following links within the same domain/path prefix. Extracts clean markdown from each page.

2. **Chunking** — Splits markdown on `h1`/`h2`/`h3` headings to preserve logical structure. Builds breadcrumb heading paths (e.g., `std > HashMap > insert`). Sub-splits oversized sections at paragraph boundaries with overlap. Tags chunks with `has_code=true` if they contain fenced code blocks.

3. **Embedding & Storage** — Embeds chunks using `all-MiniLM-L6-v2` (384-dim vectors via `sentence-transformers`), then upserts them into ChromaDB collections (one collection per programming language).

4. **Retrieval** — Embeds the user's query, performs filtered similarity search in ChromaDB, and returns formatted results with source URLs and context.

### Incremental Updates

Re-running an ingestion for the same library is efficient:
- Pages are hashed (MD5) and compared against stored hashes
- Unchanged pages are skipped entirely
- Pages removed from the site have their chunks automatically deleted

## Installation

### Prerequisites

- Python 3.10+
- [`uv`](https://github.com/astral-sh/uv) package manager

### Setup

```bash
git clone https://github.com/akshaysadanand/api-docs-mcp.git
cd api-docs-mcp
uv sync
```

## Usage

### CLI

The project ships with a `api-docs-mcp` command-line tool for managing documentation indexes.

#### Index a Documentation Site

```bash
# Index Rust standard library docs
uv run api-docs-mcp add "https://doc.rust-lang.org/std/" --language rust --library std

# Index Python docs with custom limits
uv run api-docs-mcp add "https://docs.python.org/3/library/" \
  --language python --library stdlib --max-pages 100 --max-depth 2

# Index with a specific version
uv run api-docs-mcp add "https://numpy.org/doc/stable/reference/" \
  --language python --library numpy --version 1.26
```

#### Search Indexed Documentation

```bash
# Search across all indexed docs for a language
uv run api-docs-mcp search "how to parse JSON" --language python

# Search within a specific library
uv run api-docs-mcp search "HashMap insert or update" --language rust --library std

# Get more results (default: 5)
uv run api-docs-mcp search "async await coroutine" --language kotlin -k 10
```

#### List Indexed Sources

```bash
# List all indexed sources
uv run api-docs-mcp list

# Filter by language
uv run api-docs-mcp list --language rust
```

#### Remove Indexed Documentation

```bash
uv run api-docs-mcp remove --language rust --library tokio
```

#### CLI Reference

| Command | Description | Key Arguments |
|---------|-------------|---------------|
| `add` | Crawl and index a documentation URL | `URL`, `-l/--language`, `-L/--library`, `-v/--version`, `--max-pages`, `--max-depth` |
| `search` | Search indexed docs semantically | `QUERY`, `-l/--language`, `-L/--library`, `-k/--top-k` |
| `list` | List all indexed sources | `-l/--language` (optional filter) |
| `remove` | Remove indexed docs for a library | `-l/--language`, `-L/--library` |

### MCP Server

Configure the MCP server in your AI assistant's settings to enable documentation search from within your editor.

#### VS Code (GitHub Copilot)

Add to your `.vscode/mcp.json` or user MCP settings:

```json
{
  "inputs": [],
  "servers": {
    "api-docs-mcp": {
      "command": "bash",
      "args": ["<path-to-project>/start.sh"]
    }
  }
}
```

Or start the server directly:

```bash
uv run api-docs-mcp serve
```

#### Claude Desktop

Add to your `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "api-docs-mcp": {
      "command": "uv",
      "args": ["run", "api-docs-mcp", "serve"],
      "cwd": "<path-to-project>"
    }
  }
}
```

#### Available MCP Tools

| Tool | Description | Parameters |
|------|-------------|------------|
| `search_documentation` | Semantic search across indexed docs | `query`, `language`, `library` (optional), `top_k` (default: 5) |
| `add_documentation` | Crawl and index a new doc site | `url`, `language`, `library`, `version`, `max_pages`, `max_depth` |
| `list_sources` | List all indexed sources with statistics | `language` (optional filter) |
| `get_code_examples` | Search specifically for code snippets | `query`, `language`, `library` (optional), `top_k` |
| `lookup_api_symbol` | Exact-match symbol lookup (e.g., `HashMap::insert`) | `symbol`, `language`, `library` (optional) |

## Storage

- **Database:** ChromaDB (persistent, disk-based)
- **Default location:** `~/.local/share/api-docs-mcp/` (follows `XDG_DATA_HOME`)
- **Structure:** One ChromaDB collection per programming language
- **Chunk metadata:** `library`, `version`, `source_url`, `page_hash`, `heading_path`, `has_code`, `chunk_index`
- **Embeddings:** 384-dimensional normalized vectors from `all-MiniLM-L6-v2` (~80MB model)

## Configuration

| Option | Default | Description |
|--------|---------|-------------|
| `--max-pages` | 200 | Maximum pages to crawl per run |
| `--max-depth` | 3 | Maximum link depth from start URL |
| `--version` | `latest` | Documentation version string |
| `--top-k` | 5 | Number of search results (max: 20) |
| Embedding batch size | 32 | Chunks processed per embedding batch |
| Upsert batch size | 5000 | Chunks upserted to ChromaDB per batch |

## Project Structure

```
api-docs-mcp/
├── pyproject.toml              # Project metadata & dependencies
├── start.sh                    # MCP server startup script
├── api_docs_mcp/
│   ├── cli.py                  # Click-based CLI commands
│   ├── server.py               # MCP server (stdio transport)
│   ├── embeddings/
│   │   └── model.py            # Sentence-transformer embedding model
│   ├── ingestion/
│   │   ├── crawler.py          # Headless browser web crawler
│   │   ├── chunker.py          # Markdown-aware document chunking
│   │   └── processor.py        # Orchestration & incremental updates
│   ├── retrieval/
│   │   └── engine.py           # Semantic search engine
│   └── storage/
│       └── vector_store.py     # ChromaDB vector store wrapper
├── tests/
│   ├── test_vector_store.py
│   └── test_get_metadata.py
└── docs/
    └── implementation_plan.md
```

## Dependencies

| Package | Purpose |
|---------|---------|
| [`mcp`](https://github.com/modelcontextprotocol/python-sdk) | MCP server framework (stdio transport) |
| [`click`](https://click.palletsprojects.com/) | CLI framework |
| [`chromadb`](https://www.trychroma.com/) | Vector database for persistent storage |
| [`sentence-transformers`](https://sbert.net/) | Text embedding model (`all-MiniLM-L6-v2`) |
| [`crawl4ai`](https://github.com/unclecode/crawl4ai) | Headless browser web crawler with markdown extraction |

## Examples

### Indexing Multiple Libraries

```bash
# Rust ecosystem
uv run api-docs-mcp add "https://doc.rust-lang.org/std/" -l rust -L std
uv run api-docs-mcp add "https://docs.rs/tokio/latest/tokio/" -l rust -L tokio

# Python ecosystem
uv run api-docs-mcp add "https://docs.python.org/3/library/" -l python -L stdlib --max-pages 150
uv run api-docs-mcp add "https://numpy.org/doc/stable/reference/" -l python -L numpy

# Kotlin
uv run api-docs-mcp add "https://kotlinlang.org/docs/home.html" -l kotlin -L standard --max-pages 200
```

### Typical Workflow

1. Index the documentation you work with most frequently
2. Configure the MCP server in your editor
3. Ask your AI assistant questions like:
   - "How do I use `HashMap::entry` in Rust?"
   - "Show me examples of pandas DataFrame filtering"
   - "What's the Kotlin coroutine flow API?"

## License

This project is open-sourced under the [MIT License](LICENSE)