web-docs-mcp

Local MCP server for Kilo Code / Cline / Claude Desktop — free web search + library docs fetcher with local-first caching.

Features

🔍 web_search — Free DuckDuckGo HTML search (no API key required)
🌐 fetch_url — Fetch any URL and convert to clean markdown (cached locally)
📚 lib_docs — Auto-fetch README/docs from npm, PyPI, crates.io, Go, or GitHub
📖 search_docs — Language & API documentation search biased to official docs (MDN, docs.python.org, etc.)
📂 list_docs — Browse and search your local docs/ folder first (local-first approach)

Installation

npm install
npm run build

Usage

As MCP Server

Add to your MCP client configuration (e.g., claude_desktop_config.json):

{
  "mcpServers": {
    "web-docs-mcp": {
      "command": "node",
      "args": ["/path/to/web-docs-mcp/build/index.js"],
      "env": {
        "DOCS_DIR": "/path/to/your/docs",
        "CACHE_TTL_HOURS": "168",
        "DEFAULT_SAVE_TO_DOCS": "true"
      }
    }
  }
}

Direct Execution

# Development mode
npm run dev

# Production mode
npm run start

# Build
npm run build

# Clean build artifacts
npm run clean

Configuration

All settings are configurable via environment variables:

Variable	Default	Description
`DOCS_DIR`	`./docs`	Directory where fetched markdown docs are saved
`CACHE_DIR`	`./.cache/web-docs`	Directory for TTL cache (raw HTML + fetched markdown)
`CACHE_TTL_HOURS`	`168` (7 days)	Cache time-to-live in hours
`HTTP_TIMEOUT_MS`	`20000`	HTTP timeout per request in milliseconds
`USER_AGENT`	Chrome-like UA	User-Agent sent to upstream servers
`DEFAULT_SAVE_TO_DOCS`	`true`	Default behavior for saving docs to `docs/` folder
`DISABLE_CACHE`	`false`	Disable caching (useful for debugging)
`GITHUB_TOKEN`	none	Optional GitHub token to lift rate limits

Tools

web_search

Free web search via DuckDuckGo HTML. Returns a list of {title, url, snippet}.

{
  query: string,      // Search query
  limit?: number      // Number of results (default: 8, max: 20)
}

fetch_url

Fetch a single URL and return clean markdown. Handles HTML (via turndown+GFM), JSON (pretty-printed), and plain text.

{
  url: string,                    // URL to fetch
  save?: boolean,                 // Save to docs/ folder (default: true)
  subdir?: string                 // Subdirectory under docs/ (optional)
}

lib_docs

Fetch README/docs for a library by name. Tries npm → PyPI → crates.io → Go → GitHub automatically.

{
  name: string,                   // Library name (e.g., "react", "numpy", "serde", "owner/repo")
  save?: boolean,                 // Save to docs/libraries/ (default: true)
  subdir?: string                 // Custom subdirectory (optional)
}

search_docs

Language & API documentation search. Biases results to official docs sites. LOCAL-FIRST: searches your docs/ folder first.

{
  query: string,                  // Search query
  language?: string,              // Programming language (e.g., "python", "rust", "go")
  fetch_top?: boolean,            // Fetch full markdown of top result (default: false)
  save?: boolean,                 // Save fetched content (default: true)
  subdir?: string                 // Subdirectory under docs/api/ (optional)
}

list_docs

Browse and search the local docs/ folder. Three modes:

No args = list all saved docs
{ query } = keyword search across docs/
{ path } = read full body of a specific doc by relative path or slug

{
  query?: string,                 // Keyword search (optional)
  path?: string                   // Relative path to specific doc (optional)
}

Directory Structure

web-docs-mcp/
├── src/
│   ├── index.ts          # Main entry point
│   ├── config.ts         # Configuration & env vars
│   ├── lib/              # Core utilities
│   │   ├── anubis.ts     # Anubis PoW solver (anti-bot bypass)
│   │   ├── cache.ts      # Local caching logic
│   │   ├── ddg.ts        # DuckDuckGo search
│   │   ├── fetcher.ts    # HTTP fetching
│   │   ├── html-to-md.ts # HTML to markdown conversion
│   │   └── ...
│   └── tools/            # MCP tool implementations
│       ├── web_search.ts
│       ├── fetch_url.ts
│       ├── lib_docs.ts
│       ├── search_docs.ts
│       └── list_docs.ts
├── docs/                 # Saved documentation (created on demand)
│   ├── libraries/        # Library READMEs
│   ├── api/              # API documentation
│   ├── guides/           # Tutorials & how-tos
│   └── ...
├── .cache/               # TTL cache (auto-managed)
├── build/                # Compiled JavaScript
└── package.json

Supported Ecosystems

npm — JavaScript/TypeScript packages
PyPI — Python packages
crates.io — Rust crates
pkg.go.dev — Go modules
GitHub — Any repository (owner/repo format)

Local-First Approach

This server implements a local-first strategy:

All fetched content is cached with configurable TTL
search_docs checks your local docs/ folder before going to the web
Subsequent calls return cached results in <5ms instead of re-fetching
Perfect for offline work or rate-limited environments

Requirements

Node.js >= 18.17
npm or yarn

License

MIT

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Troubleshooting

Empty search results from DuckDuckGo

The User-Agent might be blocked. Try setting a custom one:

export USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

Rate limiting on GitHub

Add a GitHub token to lift the 60 req/hour anonymous limit:

export GITHUB_TOKEN=your_token_here

Cache issues

To disable cache temporarily:

export DISABLE_CACHE=true

Or clean the cache:

npm run clean