Universal Docs MCP Server
README.md
# Universal Docs MCP Server — System Documentation
> An MCP server that gives AI assistants access to live, searchable documentation for any programming library. No more hallucinated APIs.
| Field | Value |
|-------|-------|
| Author | Kudakwashe Chikumbu |
| Date | 30 June 2026 |
| Stack | Python, FastMCP, httpx, BeautifulSoup |
| Transport | STDIO + HTTP |
| Version | 1.0 |
---
## Table of Contents
1. [Problem Statement](#01--problem-statement)
2. [Solution Overview](#02--solution-overview)
3. [Architecture](#03--architecture)
4. [MCP Tools (API Surface)](#04--mcp-tools-api-surface)
5. [Data Sources & Scraping Strategy](#05--data-sources--scraping-strategy)
6. [Caching Strategy](#06--caching-strategy)
7. [Project Structure](#07--project-structure)
8. [Tech Stack](#08--tech-stack)
9. [Configuration](#09--configuration)
10. [Error Handling](#10--error-handling)
11. [Deployment](#11--deployment)
12. [Future Roadmap](#12--future-roadmap)
---
## 01 — Problem Statement
AI coding assistants (Claude, Copilot, Cursor) hallucinate library APIs because their training data has a cutoff date. When a developer asks about a recently released function, updated parameter, or deprecated method, the AI confidently provides wrong information.
Current workarounds are all bad:
- **Manual copy-paste:** Developers paste documentation into the chat context, wasting tokens and time.
- **RAG pipelines:** Complex infrastructure (vector DBs, embeddings, chunking) for what should be a simple lookup.
- **Trust the AI:** Developers accept hallucinated APIs, ship bugs, then spend 30 minutes debugging a function signature that doesn't exist.
> âš ï¸ **The cost:** According to community data, developers waste 15-30 minutes per hallucinated API error. With AI usage at 3-5+ sessions per day, that's potentially 1-2 hours of wasted debugging per developer per week.
---
## 02 — Solution Overview
Universal Docs MCP is a server that sits between the AI client and live documentation sources. When an AI needs to reference a library's API, it calls our MCP tools instead of relying on stale training data.
### Core Value Proposition
- **Live data:** Always returns current documentation, not training-cutoff snapshots.
- **Universal:** Works with any library from PyPI, npm, crates.io, Go pkg, or any docs site.
- **Structured:** Returns clean, parsed function signatures, parameters, return types, and examples (not raw HTML).
- **Cached:** Doesn't re-scrape on every request. Intelligent TTL-based caching.
- **Client-agnostic:** Works with Claude Desktop, VS Code Copilot, Cursor, or any MCP client.
### What This Is Not
- Not a vector database or RAG system (no embeddings, no similarity search)
- Not a documentation hosting platform
- Not a replacement for reading docs (it's a lookup tool, not a tutorial engine)
---
## 03 — Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ MCP CLIENTS │
│ Claude Desktop │ VS Code Copilot │ Cursor │ Custom │
└────────┬──────────┴─────────┬─────────┴────┬─────┴──────────────┘
│ │ │
│ STDIO or HTTP (SSE) │
│ │ │
┌────────▼────────────────────▼──────────────▼────────────────────┐
│ UNIVERSAL DOCS MCP SERVER │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
│ │ Tool Router │ │ Query Parser│ │ Response Formatter │ │
│ └──────┬───────┘ └──────┬───────┘ └────────────┬───────────┘ │
│ │ │ │ │
│ ┌──────▼──────────────────▼───────────────────────▼───────────┐ │
│ │ DOCS ENGINE │ │
│ │ │ │
│ │ ┌─────────────┐ ┌──────────────┐ ┌───────────────────┐ │ │
│ │ │ Scraper │ │ Parser │ │ Cache (SQLite) │ │ │
│ │ │ (httpx + │ │ (BS4 + │ │ TTL: 6 hours │ │ │
│ │ │ async) │ │ custom) │ │ LRU: 1000 items │ │ │
│ │ └──────┬──────┘ └──────┬───────┘ └───────────────────┘ │ │
│ │ │ │ │ │
│ └─────────┼─────────────────┼──────────────────────────────────┘ │
│ │ │ │
└────────────┼─────────────────┼────────────────────────────────────┘
│ │
┌────────────▼─────────────────▼────────────────────────────────────┐
│ DOCUMENTATION SOURCES │
│ │
│ PyPI/ReadTheDocs │ npm/MDN │ Docs Sites │ GitHub READMEs │
└────────────────────┴───────────┴──────────────┴──────────────────┘
```
### Component Responsibilities
| Component | Responsibility | Key Decisions |
|-----------|---------------|---------------|
| **Tool Router** | Maps incoming MCP tool calls to the correct handler | Stateless, pure function dispatch |
| **Query Parser** | Normalizes library names, versions, function paths | Handles aliases (e.g. "bs4" → "beautifulsoup4") |
| **Docs Engine** | Orchestrates fetch → parse → cache → return | Async pipeline, fail-fast on unknown libraries |
| **Scraper** | Fetches raw HTML from documentation sites | httpx async, respects robots.txt, rate-limited |
| **Parser** | Extracts structured data from HTML docs | Per-source parsing strategies (registry pattern) |
| **Cache** | Stores parsed docs, avoids redundant scraping | SQLite for persistence, in-memory LRU for hot path |
| **Response Formatter** | Shapes output for minimal token usage by AI | Concise, structured, no HTML in output |
---
## 04 — MCP Tools (API Surface)
The server exposes 5 core tools. Each is designed to answer a specific class of developer question with minimal token waste.
### Tool 1: `search_docs`
Full-text search across a library's documentation. The "I know it exists but can't remember the name" tool.
**Schema:**
```python
@mcp.tool()
async def search_docs(
library: str,
query: str,
version: str = "latest",
max_results: int = 5
) -> list[dict]:
"""
Search documentation for a library by keyword.
Args:
library: Package name (e.g. "fastapi", "pandas", "react")
query: Search term (e.g. "async middleware", "groupby aggregate")
version: Specific version or "latest" (default)
max_results: Maximum results to return (1-10)
Returns:
List of matching doc entries with name, signature,
short description, and source URL.
"""
```
**Example Call:**
```python
search_docs(library="fastapi", query="dependency injection", max_results=3)
```
**Example Response:**
```json
[
{
"name": "Depends",
"module": "fastapi.params",
"signature": "Depends(dependency, *, use_cache=True)",
"description": "Declare a FastAPI dependency. The dependency callable will be called with the same parameters as the endpoint.",
"url": "https://fastapi.tiangolo.com/tutorial/dependencies/"
}
]
```
---
### Tool 2: `get_function_reference`
Precise lookup of a specific function, class, or method. Returns full signature, all parameters with types and defaults, return type, and usage examples.
**Schema:**
```python
@mcp.tool()
async def get_function_reference(
library: str,
function_path: str,
version: str = "latest",
include_examples: bool = True
) -> dict:
"""
Get complete reference for a specific function or class.
Args:
library: Package name
function_path: Dotted path (e.g. "DataFrame.merge", "Router.get")
version: Specific version or "latest"
include_examples: Whether to include code examples
Returns:
Full function reference: signature, params, return type,
description, examples, deprecation warnings.
"""
```
**Example Response:**
```json
{
"name": "merge",
"module": "pandas.DataFrame",
"signature": "DataFrame.merge(right, how='inner', on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=False, suffixes=('_x', '_y'), copy=None, indicator=False, validate=None)",
"parameters": [
{"name": "right", "type": "DataFrame | Series", "required": true, "description": "Object to merge with"},
{"name": "how", "type": "str", "default": "'inner'", "description": "Type of merge: 'left', 'right', 'outer', 'inner', 'cross'"}
],
"returns": {"type": "DataFrame", "description": "A DataFrame of the two merged objects"},
"examples": ["df1.merge(df2, on='key', how='left')"],
"deprecated": false,
"added_in_version": "0.19.0",
"url": "https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html"
}
```
---
### Tool 3: `get_module_overview`
High-level overview of a module or subpackage. Returns all public classes, functions, and constants with one-line descriptions.
**Schema:**
```python
@mcp.tool()
async def get_module_overview(
library: str,
module_path: str = "",
version: str = "latest"
) -> dict:
"""
List all public symbols in a module with brief descriptions.
Args:
library: Package name
module_path: Submodule path (empty = top-level package)
version: Specific version or "latest"
Returns:
Module overview with classes, functions, constants listed.
"""
```
---
### Tool 4: `get_changelog`
What changed between versions? Critical for migration and debugging "this worked yesterday" situations.
**Schema:**
```python
@mcp.tool()
async def get_changelog(
library: str,
from_version: str = None,
to_version: str = "latest",
breaking_only: bool = False
) -> dict:
"""
Get changelog entries between two versions.
Args:
library: Package name
from_version: Starting version (None = last 3 releases)
to_version: Ending version (default: latest)
breaking_only: Only show breaking changes
Returns:
Changelog with version, date, breaking changes,
new features, bug fixes, and deprecations.
"""
```
---
### Tool 5: `get_install_info`
Quick package metadata: latest version, install command, dependencies, Python/Node version requirements.
**Schema:**
```python
@mcp.tool()
async def get_install_info(
library: str,
ecosystem: str = "auto"
) -> dict:
"""
Get installation info and package metadata.
Args:
library: Package name
ecosystem: "pypi", "npm", "cargo", "go", or "auto" (detect)
Returns:
Install command, latest version, license, dependencies,
runtime requirements, and repository URL.
"""
```
---
## 05 — Data Sources & Scraping Strategy
Documentation lives in different places depending on the ecosystem. The server uses a registry of "source adapters" that know how to fetch and parse docs from each source type.
### Source Priority (per ecosystem)
| Ecosystem | Primary Source | Fallback | Metadata |
|-----------|---------------|----------|----------|
| Python | ReadTheDocs / official docs site | PyPI description, GitHub README | PyPI JSON API |
| JavaScript | Official docs site / MDN | npm README, GitHub README | npm registry API |
| Rust | docs.rs | GitHub README | crates.io API |
| Go | pkg.go.dev | GitHub README | pkg.go.dev API |
### Source Adapter Pattern
```python
class BaseSourceAdapter:
"""Base class for documentation source adapters."""
async def can_handle(self, library: str) -> bool:
"""Return True if this adapter can fetch docs for this library."""
...
async def fetch_search(self, library: str, query: str) -> list[dict]:
"""Search documentation and return structured results."""
...
async def fetch_reference(self, library: str, path: str) -> dict:
"""Fetch a specific function/class reference."""
...
async def fetch_changelog(self, library: str) -> list[dict]:
"""Fetch version changelog."""
...
class PyPIAdapter(BaseSourceAdapter):
"""Handles Python packages via PyPI + ReadTheDocs."""
...
class NpmAdapter(BaseSourceAdapter):
"""Handles JavaScript packages via npm + official docs."""
...
class DocsRsAdapter(BaseSourceAdapter):
"""Handles Rust crates via docs.rs."""
...
```
### Scraping Rules
- Respect `robots.txt` on all domains
- Rate limit: max 2 requests/second per domain
- Set a descriptive User-Agent: `UniversalDocsMCP/1.0 (docs-lookup; +github.com/you/repo)`
- Prefer JSON APIs where available (PyPI, npm, crates.io) over scraping
- Cache aggressively to minimize requests
- Graceful degradation: if scraping fails, return metadata-only response with URL
> âš ï¸ **Important:** Start with PyPI + ReadTheDocs support only. Add npm, Rust, Go adapters incrementally. A working Python docs server is more valuable than a broken multi-ecosystem one.
---
## 06 — Caching Strategy
Documentation doesn't change every minute. A smart caching layer reduces network calls by 90%+ while keeping data reasonably fresh.
### Two-Layer Cache
| Layer | Storage | TTL | Capacity | Purpose |
|-------|---------|-----|----------|---------|
| **L1: Hot** | In-memory (dict) | 30 minutes | 500 entries (LRU) | Instant response for repeated queries in same session |
| **L2: Warm** | SQLite (disk) | 6 hours | 10,000 entries | Persist across server restarts, serve stale if source is down |
### Cache Key Structure
```
# Format: {ecosystem}:{library}:{version}:{tool}:{query_hash}
# Examples:
"pypi:fastapi:0.115.0:reference:sha256_abc123"
"pypi:pandas:latest:search:sha256_def456"
"npm:react:19.0.0:changelog:sha256_ghi789"
```
### Cache Invalidation Rules
- **Version-pinned queries:** Long TTL (24 hours). Docs for v2.1.0 won't change.
- **"latest" queries:** Short TTL (6 hours). Re-check what "latest" means.
- **Search queries:** Medium TTL (6 hours). New content might match.
- **Install info:** Short TTL (1 hour). Versions release frequently.
- **Stale-while-revalidate:** If source is unreachable, serve stale cache entry with a warning flag.
### SQLite Schema
```sql
CREATE TABLE docs_cache (
cache_key TEXT PRIMARY KEY,
data TEXT NOT NULL, -- JSON blob
created_at INTEGER NOT NULL, -- Unix timestamp
expires_at INTEGER NOT NULL, -- Unix timestamp
source_url TEXT, -- Where it was fetched from
hit_count INTEGER DEFAULT 0 -- For LRU eviction
);
CREATE INDEX idx_expires ON docs_cache(expires_at);
CREATE INDEX idx_hits ON docs_cache(hit_count);
```
---
## 07 — Project Structure
```
universal-docs-mcp/
├── pyproject.toml # Project config, dependencies
├── README.md
├── .env.example # Environment template
├── Dockerfile
├── src/
│ ├── __init__.py
│ ├── server.py # MCP server entry point & tool definitions
│ ├── config.py # Settings, env vars, defaults
│ ├── engine/
│ │ ├── __init__.py
│ │ ├── docs_engine.py # Orchestrator: fetch → parse → cache → return
│ │ ├── query_parser.py # Normalize library names, resolve aliases
│ │ └── formatter.py # Shape responses for minimal tokens
│ ├── adapters/
│ │ ├── __init__.py
│ │ ├── base.py # BaseSourceAdapter ABC
│ │ ├── pypi.py # Python/PyPI/ReadTheDocs adapter
│ │ ├── npm.py # JavaScript/npm adapter
│ │ └── github.py # GitHub README fallback adapter
│ ├── cache/
│ │ ├── __init__.py
│ │ ├── memory_cache.py # L1 in-memory LRU
│ │ └── sqlite_cache.py # L2 persistent SQLite
│ └── utils/
│ ├── __init__.py
│ ├── http_client.py # Shared httpx client with rate limiting
│ └── html_parser.py # HTML → structured data utilities
├── tests/
│ ├── test_search.py
│ ├── test_reference.py
│ ├── test_cache.py
│ └── test_adapters.py
└── .vscode/
└── mcp.json # Local MCP config for testing
```
---
## 08 — Tech Stack
| Layer | Technology | Why |
|-------|-----------|-----|
| MCP Framework | `fastmcp >= 3.4` | Decorator-based, handles protocol plumbing |
| HTTP Client | `httpx[http2]` | Async, HTTP/2, connection pooling, timeout control |
| HTML Parsing | `beautifulsoup4` + `lxml` | Fast, forgiving HTML parsing |
| Cache (L2) | `aiosqlite` | Async SQLite, zero infrastructure, file-based |
| Config | `pydantic-settings` | Typed config from env vars with validation |
| Rate Limiting | `asyncio.Semaphore` + custom | Per-domain rate limits, no external dependency |
| Testing | `pytest` + `pytest-asyncio` | Async test support, fixtures for mocked responses |
| HTTP Server (optional) | `uvicorn` + `fastapi` | Only needed for HTTP transport deployment |
| Package Manager | `uv` | Fast dependency resolution, lockfile support |
### Install Command
```bash
uv add fastmcp httpx[http2] beautifulsoup4 lxml aiosqlite pydantic-settings
```
### Dev Dependencies
```bash
uv add --dev pytest pytest-asyncio respx ruff
```
---
## 09 — Configuration
### `.env.example`
```env
# Cache
CACHE_DIR=./data # SQLite file location
CACHE_TTL_HOURS=6 # Default TTL for cached docs
CACHE_MAX_MEMORY_ENTRIES=500 # L1 in-memory cache size
# Network
REQUEST_TIMEOUT_SECONDS=10 # Per-request timeout
MAX_CONCURRENT_REQUESTS=5 # Global concurrency limit
RATE_LIMIT_PER_DOMAIN=2 # Requests per second per domain
USER_AGENT="UniversalDocsMCP/1.0"
# Server
TRANSPORT=stdio # "stdio" or "http"
HTTP_HOST=0.0.0.0 # Only for HTTP transport
HTTP_PORT=8000 # Only for HTTP transport
# Optional: GitHub token for higher rate limits on README fetches
GITHUB_TOKEN= # ghp_xxxxxxxxxxxx
```
### `src/config.py`
```python
from pydantic_settings import BaseSettings
class Settings(BaseSettings):
# Cache
cache_dir: str = "./data"
cache_ttl_hours: int = 6
cache_max_memory_entries: int = 500
# Network
request_timeout_seconds: int = 10
max_concurrent_requests: int = 5
rate_limit_per_domain: int = 2
user_agent: str = "UniversalDocsMCP/1.0"
# Server
transport: str = "stdio"
http_host: str = "0.0.0.0"
http_port: int = 8000
# Optional
github_token: str | None = None
class Config:
env_file = ".env"
settings = Settings()
```
---
## 10 — Error Handling
The server must never crash or return raw exceptions to the AI client. Every failure mode has a graceful response.
| Failure | Response to Client | Internal Action |
|---------|-------------------|-----------------|
| Library not found | `{"error": "not_found", "message": "Library 'xyz' not found in PyPI or npm", "suggestion": "Did you mean 'xyzlib'?"}` | Check aliases, suggest similar names |
| Function path invalid | `{"error": "invalid_path", "message": "No function 'Foo.baz' in library X", "available": ["Foo.bar", "Foo.bat"]}` | Fuzzy match available symbols |
| Source unreachable (timeout) | Return stale cache if available, else: `{"error": "source_unavailable", "cached_at": "...", "data": ...}` | Log, increment failure counter, use stale cache |
| Rate limited by source | Return stale cache or retry-after message | Back off, respect Retry-After header |
| Parsing failed (HTML changed) | `{"error": "parse_error", "message": "Docs structure changed, returning raw URL", "url": "..."}` | Log for adapter maintenance, return URL as fallback |
| Invalid parameters | FastMCP handles via Pydantic validation automatically | N/A |
> â„¹ï¸ **Design Principle:** Always return something useful. A URL to the docs page is better than an empty error. Stale data with a warning is better than no data.
---
## 11 — Deployment
### Local Development (STDIO)
For testing with VS Code Copilot or Claude Desktop:
**`.vscode/mcp.json`:**
```json
{
"servers": {
"UniversalDocs": {
"command": "python",
"args": ["-m", "src.server"]
}
}
}
```
**Claude Desktop config (`%APPDATA%\Claude\claude_desktop_config.json`):**
```json
{
"mcpServers": {
"UniversalDocs": {
"command": "C:/path/to/.venv/Scripts/python.exe",
"args": ["-m", "src.server"]
}
}
}
```
### Remote Deployment (HTTP via Docker)
**`Dockerfile`:**
```dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN pip install uv && uv sync --frozen
COPY src/ ./src/
COPY .env .env
ENV TRANSPORT=http
EXPOSE 8000
CMD ["python", "-m", "src.server"]
```
**`docker-compose.yml`:**
```yaml
services:
universal-docs-mcp:
build: .
ports:
- "8000:8000"
volumes:
- ./data:/app/data # Persist SQLite cache
environment:
- TRANSPORT=http
- GITHUB_TOKEN=${GITHUB_TOKEN}
restart: unless-stopped
```
### Production Checklist
- [ ] Enable HTTPS (nginx reverse proxy or cloud load balancer)
- [ ] Add authentication if exposing publicly (API key header or OAuth)
- [ ] Set up health check endpoint at `/health`
- [ ] Monitor cache hit rates (target: >80%)
- [ ] Log scraping failures for adapter maintenance
- [ ] Set memory limits on container (SQLite + LRU shouldn't exceed 512MB)
---
## 12 — Future Roadmap
### Phase 1: MVP (Week 1-2)
- [ ] Python/PyPI adapter only
- [ ] `search_docs` and `get_function_reference` tools
- [ ] SQLite cache
- [ ] STDIO transport
- [ ] Working in VS Code Copilot
### Phase 2: Expand (Week 3-4)
- [ ] Add `get_changelog` and `get_install_info` tools
- [ ] npm/JavaScript adapter
- [ ] GitHub README fallback adapter
- [ ] HTTP transport option
- [ ] Claude Desktop support confirmed
### Phase 3: Production (Week 5-6)
- [ ] Docker deployment
- [ ] Rust (docs.rs) and Go (pkg.go.dev) adapters
- [ ] Library alias database (bs4 → beautifulsoup4, etc.)
- [ ] Fuzzy search for typos in function names
- [ ] Cache analytics dashboard
### Phase 4: Community (Week 7+)
- [ ] Publish to MCP server registry
- [ ] Custom docs source configuration (point at any docs site)
- [ ] Plugin system for community-contributed adapters
- [ ] Version diff tool ("what changed between FastAPI 0.109 and 0.115?")
- [ ] Offline mode (pre-cache top 100 libraries)
> ✅ **Start here:** Phase 1 is your first PR. Get `search_docs` working for one Python library (try FastAPI, its docs are well-structured HTML). Everything else builds on top.This server cannot be deployed
Maintenance
ActivitySlowing
ResponsivenessNo issues