MCP Web Search Pro

An extended, self-hosted web research server for MCP-compatible clients. It gives an AI assistant tools for discovering web pages, extracting readable content, rendering JavaScript applications, using the Internet Archive when a page is unavailable, and reading YouTube captions.

The service is designed for local or trusted-network use. It is stateless, requires no API keys, and exposes the MCP Streamable HTTP transport on port 8082. It can run beside a lighter web-search MCP server on another port.

Runs as a separate container on port 8082 alongside the lightweight mcp-web-search (port 8081). Point clients at whichever fits the task — if pro ever feels sluggish, the lightweight one is still right there untouched.

Features

• Web search through DuckDuckGo, with result limits and region selection.

• Readable HTML extraction to Markdown with navigation and boilerplate removed.

• Automatic PDF detection and text extraction.

• Headless Chromium rendering for React, Vue, SPA, and other JS-heavy pages.

• Wayback Machine snapshots for unavailable or changed pages.

• YouTube transcripts from URLs or video IDs, with optional timestamps.

• /health endpoint for container and process checks.

• Stateless Streamable HTTP and stdio MCP transports.

Related MCP server: Basic MCP Tools

MCP tools

Use web_fetch for ordinary pages because it is faster. Use web_fetch_js when the first tool returns an empty shell or incomplete content because the site depends on client-side JavaScript.

Run with Docker

docker compose up -d --build

Verify:

curl http://localhost:8082/health      # → {"status":"ok"}

Logs / restart / stop:

docker compose logs -f
docker compose restart
docker compose down

The first build is large (~400MB) because it downloads Chromium for Playwright. Subsequent builds reuse the cached layer.

Connecting clients

Point any MCP-compatible client at:

http://<server-host>:8082/mcp

No headers or API keys are required. The server runs in stateless mode.

Run locally without Docker

Python 3.12 is recommended:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
playwright install --with-deps chromium
python server.py

Run the HTTP transport directly with:

python server.py --sse --host 127.0.0.1 --port 8082

Despite the historical --sse option name, the HTTP implementation uses the stateless MCP Streamable HTTP transport.

Performance notes

• Only web_fetch_js is heavy at runtime — it launches Chromium per call (~5–15s). The other four tools are lightweight pure-Python and stay snappy.

• shm_size: 1gb in docker-compose.yml prevents Chromium sandbox crashes inside the container.

• For routine non-JS pages, use web_fetch (fast); reserve web_fetch_js for React/Vue/SPA sites that render content with JavaScript.

Configuration

To change the port, edit both the ports: mapping and the command: in docker-compose.yml. To enable raw request/response logging for debugging, switch to the commented --verbose command: line.

Validation

This repository is intended to be pushed as source code and Docker build configuration. There is currently no automated test suite. The recommended project checks are:

git diff --check
docker compose config
docker compose up -d --build
curl http://localhost:8082/health
docker compose logs --tail=100
docker compose down

Do not publish this service directly to the public internet without adding authentication or placing it behind a trusted reverse proxy. The HTTP mode binds to 0.0.0.0, permits requests without API keys, and allows all origins; that is convenient for local MCP clients but is not an access-control layer.

Security and operating notes

• web_fetch and web_fetch_js can request arbitrary HTTP(S) URLs. Restrict network access or add URL/DNS safeguards if untrusted users can call the service.

• Keep --verbose disabled outside short local debugging sessions because it logs request and response bodies.

• Dependencies use minimum-version constraints. Pin or lock versions before relying on reproducible production builds.

Files

• server.py — the MCP server (5 tools)

• requirements.txt — mcp, httpx, beautifulsoup4, markdownify, ddgs, trafilatura, pypdf, youtube-transcript-api, playwright

• Dockerfile — Python 3.12-slim + Playwright Chromium

• docker-compose.yml — service on port 8082 with shm_size: 1gb

License

No license file is currently included. Add a license before accepting external contributions or redistributing the project.