# MCP Web Search Pro

An extended, self-hosted web research server for MCP-compatible clients. It
gives an AI assistant tools for discovering web pages, extracting readable
content, rendering JavaScript applications, using the Internet Archive when a
page is unavailable, and reading YouTube captions.

The service is designed for local or trusted-network use. It is stateless,
requires no API keys, and exposes the MCP Streamable HTTP transport on port
`8082`. It can run beside a lighter web-search MCP server on another port.

Runs as a **separate container on port 8082** alongside the lightweight
`mcp-web-search` (port 8081). Point clients at whichever fits the task — if pro
ever feels sluggish, the lightweight one is still right there untouched.

## Features

- Web search through DuckDuckGo, with result limits and region selection.
- Readable HTML extraction to Markdown with navigation and boilerplate removed.
- Automatic PDF detection and text extraction.
- Headless Chromium rendering for React, Vue, SPA, and other JS-heavy pages.
- Wayback Machine snapshots for unavailable or changed pages.
- YouTube transcripts from URLs or video IDs, with optional timestamps.
- `/health` endpoint for container and process checks.
- Stateless Streamable HTTP and stdio MCP transports.

## MCP tools

| Tool | Description |
|------|-------------|
| `web_search`         | Search the web via DuckDuckGo (ddgs). Args: `query` (required), `max_results` (1–20), `region`. |
| `web_fetch`          | Fetch a URL → Markdown. Auto-detects PDFs and extracts text. Uses trafilatura for clean article extraction (strips nav/ads/boilerplate). Args: `url`, `timeout`. |
| `web_fetch_js`       | Render a JS-heavy / SPA page with headless Chromium, then extract text. Use when `web_fetch` returns an empty/stub page. Slower (~5–15s). Args: `url`, `wait_ms` (0–15000, default 2500), `timeout`. |
| `web_fetch_archive`  | Fetch the latest (or timestamped) archived snapshot of a URL from the Wayback Machine. Retries on transient 503s. Args: `url`, `timestamp` (optional `YYYYMMDDhhmm`). |
| `youtube_transcript` | Fetch a YouTube video's transcript. Accepts a full YouTube URL or bare 11-char video id. Args: `url`, `include_timestamps` (bool, default false). |

Use `web_fetch` for ordinary pages because it is faster. Use `web_fetch_js`
when the first tool returns an empty shell or incomplete content because the
site depends on client-side JavaScript.

## Run with Docker

```bash
docker compose up -d --build
```

Verify:
```bash
curl http://localhost:8082/health      # → {"status":"ok"}
```

Logs / restart / stop:
```bash
docker compose logs -f
docker compose restart
docker compose down
```

> The first build is large (~400MB) because it downloads Chromium for
> Playwright. Subsequent builds reuse the cached layer.

## Connecting clients

Point any MCP-compatible client at:

```
http://<server-host>:8082/mcp
```

No headers or API keys are required. The server runs in stateless mode.

## Run locally without Docker

Python 3.12 is recommended:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
playwright install --with-deps chromium
python server.py
```

Run the HTTP transport directly with:

```bash
python server.py --sse --host 127.0.0.1 --port 8082
```

Despite the historical `--sse` option name, the HTTP implementation uses the
stateless MCP Streamable HTTP transport.

## Performance notes

- Only `web_fetch_js` is heavy at runtime — it launches Chromium per call
  (~5–15s). The other four tools are lightweight pure-Python and stay snappy.
- `shm_size: 1gb` in `docker-compose.yml` prevents Chromium sandbox crashes
  inside the container.
- For routine non-JS pages, use `web_fetch` (fast); reserve `web_fetch_js` for
  React/Vue/SPA sites that render content with JavaScript.

## Configuration

To change the port, edit both the `ports:` mapping and the `command:` in
`docker-compose.yml`. To enable raw request/response logging for debugging,
switch to the commented `--verbose` `command:` line.

## Validation

This repository is intended to be pushed as source code and Docker build
configuration. There is currently no automated test suite. The recommended
project checks are:

```bash
git diff --check
docker compose config
docker compose up -d --build
curl http://localhost:8082/health
docker compose logs --tail=100
docker compose down
```

Do not publish this service directly to the public internet without adding
authentication or placing it behind a trusted reverse proxy. The HTTP mode
binds to `0.0.0.0`, permits requests without API keys, and allows all origins;
that is convenient for local MCP clients but is not an access-control layer.

## Security and operating notes

- `web_fetch` and `web_fetch_js` can request arbitrary HTTP(S) URLs. Restrict
  network access or add URL/DNS safeguards if untrusted users can call the
  service.
- Keep `--verbose` disabled outside short local debugging sessions because it
  logs request and response bodies.
- Dependencies use minimum-version constraints. Pin or lock versions before
  relying on reproducible production builds.

## Files

- `server.py` — the MCP server (5 tools)
- `requirements.txt` — `mcp`, `httpx`, `beautifulsoup4`, `markdownify`, `ddgs`,
  `trafilatura`, `pypdf`, `youtube-transcript-api`, `playwright`
- `Dockerfile` — Python 3.12-slim + Playwright Chromium
- `docker-compose.yml` — service on port 8082 with `shm_size: 1gb`

## License

No license file is currently included. Add a license before accepting external
contributions or redistributing the project.