MCP Web Search Pro

An extended, self-hosted web research server for MCP-compatible clients. It gives an AI assistant tools for discovering web pages, extracting readable content, rendering JavaScript applications, using the Internet Archive when a page is unavailable, and reading YouTube captions.

The service is designed for local or trusted-network use. It is stateless, requires no API keys, and exposes the MCP Streamable HTTP transport on port 8082. It can run beside a lighter web-search MCP server on another port.

Runs as a separate container on port 8082 alongside the lightweight mcp-web-search (port 8081). Point clients at whichever fits the task — if pro ever feels sluggish, the lightweight one is still right there untouched.

Features

Web search through DuckDuckGo, with result limits and region selection.
Readable HTML extraction to Markdown with navigation and boilerplate removed.
Automatic PDF detection and text extraction.
Headless Chromium rendering for React, Vue, SPA, and other JS-heavy pages.
Wayback Machine snapshots for unavailable or changed pages.
YouTube transcripts from URLs or video IDs, with optional timestamps.
/health endpoint for container and process checks.
Stateless Streamable HTTP and stdio MCP transports.

MCP tools

Tool	Description
`web_search`	Search the web via DuckDuckGo (ddgs). Args: `query` (required), `max_results` (1–20), `region`.
`web_fetch`	Fetch a URL → Markdown. Auto-detects PDFs and extracts text. Uses trafilatura for clean article extraction (strips nav/ads/boilerplate). Args: `url`, `timeout`.
`web_fetch_js`	Render a JS-heavy / SPA page with headless Chromium, then extract text. Use when `web_fetch` returns an empty/stub page. Slower (~5–15s). Args: `url`, `wait_ms` (0–15000, default 2500), `timeout`.
`web_fetch_archive`	Fetch the latest (or timestamped) archived snapshot of a URL from the Wayback Machine. Retries on transient 503s. Args: `url`, `timestamp` (optional `YYYYMMDDhhmm`).
`youtube_transcript`	Fetch a YouTube video's transcript. Accepts a full YouTube URL or bare 11-char video id. Args: `url`, `include_timestamps` (bool, default false).

Use web_fetch for ordinary pages because it is faster. Use web_fetch_js when the first tool returns an empty shell or incomplete content because the site depends on client-side JavaScript.

Run with Docker

docker compose up -d --build

Verify:

curl http://localhost:8082/health      # → {"status":"ok"}

Logs / restart / stop:

docker compose logs -f
docker compose restart
docker compose down

The first build is large (~400MB) because it downloads Chromium for Playwright. Subsequent builds reuse the cached layer.

Connecting clients

Point any MCP-compatible client at:

http://<server-host>:8082/mcp

No headers or API keys are required. The server runs in stateless mode.

Run locally without Docker

Python 3.12 is recommended:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
playwright install --with-deps chromium
python server.py

Run the HTTP transport directly with:

python server.py --sse --host 127.0.0.1 --port 8082

Despite the historical --sse option name, the HTTP implementation uses the stateless MCP Streamable HTTP transport.

Performance notes

Only web_fetch_js is heavy at runtime — it launches Chromium per call (~5–15s). The other four tools are lightweight pure-Python and stay snappy.
shm_size: 1gb in docker-compose.yml prevents Chromium sandbox crashes inside the container.
For routine non-JS pages, use web_fetch (fast); reserve web_fetch_js for React/Vue/SPA sites that render content with JavaScript.

Configuration

To change the port, edit both the ports: mapping and the command: in docker-compose.yml. To enable raw request/response logging for debugging, switch to the commented --verbose command: line.

Validation

This repository is intended to be pushed as source code and Docker build configuration. There is currently no automated test suite. The recommended project checks are:

git diff --check
docker compose config
docker compose up -d --build
curl http://localhost:8082/health
docker compose logs --tail=100
docker compose down

Do not publish this service directly to the public internet without adding authentication or placing it behind a trusted reverse proxy. The HTTP mode binds to 0.0.0.0, permits requests without API keys, and allows all origins; that is convenient for local MCP clients but is not an access-control layer.

Security and operating notes

web_fetch and web_fetch_js can request arbitrary HTTP(S) URLs. Restrict network access or add URL/DNS safeguards if untrusted users can call the service.
Keep --verbose disabled outside short local debugging sessions because it logs request and response bodies.
Dependencies use minimum-version constraints. Pin or lock versions before relying on reproducible production builds.

Files

server.py — the MCP server (5 tools)
requirements.txt — mcp, httpx, beautifulsoup4, markdownify, ddgs, trafilatura, pypdf, youtube-transcript-api, playwright
Dockerfile — Python 3.12-slim + Playwright Chromium
docker-compose.yml — service on port 8082 with shm_size: 1gb

License

No license file is currently included. Add a license before accepting external contributions or redistributing the project.