# webresearcher-mcp-server

A light, standalone **web-research MCP server**: DuckDuckGo search, headless
Camoufox page scraping (a hardened open-source Firefox), and same-domain
crawling — zero Firecrawl, no API keys, no per-client browser download (in
Docker mode).

It exposes four LLM-facing tools over MCP; the host LLM does the synthesis:

| Tool | What it does |
|---|---|
| `web_search(query, limit=5)` | DuckDuckGo via [`ddgs`](https://github.com/hardkoded/ddgs) (browser-grade TLS). Numbered results: title, URL, snippet. |
| `web_scrape(url, max_chars=20000)` | [Camoufox](https://camoufox.com) — a hardened open-source Firefox. Renders JavaScript and gets past cookie-consent banners (Yahoo/OneTrust/Cookiebot) that a bare HTTP fetch skips; degrades to a "rely on search" note when a page is protected or unreachable. `max_chars=0` = full text. |
| `web_crawl(url, limit=10)` | Starts a background same-domain BFS crawl (one browser, wall-detected per page). Returns a crawl id immediately — never blocks. |
| `web_crawl_status(crawl_id, max_chars=20000)` | Crawl progress; on completion, per-page URLs + content (0 = full text). |

Failures are graceful by design: a blocked page or outage returns a short
`SCRAPE UNAVAILABLE` / `SEARCH UNAVAILABLE` note (never an exception), and the
note tells the LLM not to retry the same URL/query.

## Why not just WebFetch / WebSearch?

This is **not** a replacement for your IDE's built-in fetch/search — it is the
tool for JS-rendered, consent-gated, and multi-page research:

- **`web_scrape` reads pages a bare HTTP fetch can't.** It runs a real browser
  (Camoufox), so JavaScript renders and cookie-consent overlays are handled —
  where the built-in fetch returns an empty shell, this returns the content.
  When a page is protected or unreachable it degrades to a clear "rely on
  search" note instead of failing.
- **`web_crawl` has no built-in equivalent** (multi-page BFS + per-page
  blocked-page detection).
- **`web_search`** is free and independent of the harness's search
  availability; built-in search has the better generic index — use both.
- Plain public static pages: the built-in fetch is fine, no need to route
  everything through this server.

A typical session: `web_search("BOX stock news")` → `web_scrape` the 2–4 best
results → optional `web_crawl` of the newsroom → synthesize with citations.

## Quick start (uvx — no Docker)

Run it straight from PyPI. `uvx` auto-installs the package (and its `mcp`
extra) into an isolated env and runs the stdio server as an ordinary child
process — no Docker, no install step, and it's cleaned up when your session
ends:

```bash
uvx "webresearcher-mcp-server[mcp]" --help    # sanity check
uvx "webresearcher-mcp-server[mcp]"          # stdio server (default transport)
```

The Camoufox browser (used by `web_scrape`/`web_crawl`) is downloaded on
first use (~5 min one-time) or pre-fetched with `python -m camoufox fetch`.

**Shared HTTP server** (optional — one detached process shared by every
IDE/session so pacing + cache are global; the only Docker path we still use):

```bash
docker build -t webresearch-mcp .     # or: docker compose up -d
docker run -d --name webresearch-mcp -p 8001:8001 webresearch-mcp \
  --transport streamable-http --host 0.0.0.0 --port 8001
```

(healthcheck: `curl -fs -X POST http://localhost:8001/mcp -H
'Content-Type: application/json' -H 'Accept: application/json,
text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'` →
lists the four tools)

## Integrating with Claude Code / Cursor

### Claude Code

**uvx (recommended — no Docker):** add to your project `.mcp.json`
(or `claude mcp add`). First use auto-installs the package into an isolated
env (cached afterward); the server runs as an ordinary child process and is
cleaned up when the session ends:

```json
{ "mcpServers": {
    "webresearch": {
      "command": "uvx",
      "args": ["webresearcher-mcp-server[mcp]", "--transport", "stdio"]
    } } }
```

> **Avoid the `docker run -i --rm` per-session shape.** It leaks orphaned
> containers: Claude Code closes the child's stdin pipe on exit, the container
> (owned by the Docker daemon) never receives a stop signal, so `--rm` never
> fires and stray containers pile up. Known issue:
> [anthropics/claude-code#29058](https://github.com/anthropics/claude-code/issues/29058).

**Docker-HTTP (one detached container shared by all sessions):** run the
shared container above, then:

```json
{ "mcpServers": { "webresearch": { "type": "http",
                                    "url": "http://localhost:8001/mcp" } } }
```

**Plain venv (no `uv`/`uvx` on the box):** install the package in a venv, then:

```json
{ "mcpServers": { "webresearch": {
      "command": "/path/to/.venv/bin/python",
      "args": ["-m", "webresearch_mcp"] } } }
```

### Cursor

Same shapes in `~/.cursor/mcp.json` (user) or `.cursor/mcp.json` (project).
uvx (no Docker):

```json
{ "mcpServers": { "webresearch": {
      "type": "stdio",
      "command": "uvx",
      "args": ["webresearcher-mcp-server[mcp]", "--transport", "stdio"] } } }
```

or the `{"url": "http://localhost:8001/mcp"}` form for the shared HTTP
container. See [docs/integration.md](docs/integration.md) for the full
walkthrough, including the optional **cross-IDE plugin** (`plugin/`): one
directory, two manifests (Claude Code + Cursor), bundling the stdio server +
an always-on research rule + a `/research` command + a `web-researcher`
subagent — no Docker needed.

## Plain Python (no `uv`/`uvx`)

```bash
python3 -m venv .venv && . .venv/bin/activate
pip install "webresearcher-mcp-server[mcp]"   # or: pip install -e ".[mcp]" from a clone
python -m camoufox fetch          # one-time browser download
webresearcher-mcp-server                   # stdio server
webresearcher-mcp-server --transport streamable-http --port 8001
```

## Configuration (env vars)

| Variable | Default | Effect |
|---|---|---|
| `WEBRESEARCH_MIN_CALL_INTERVAL_S` | `10` | Paces the *start* of search/scrape calls (polite rate-limiting). |
| `WEBRESEARCH_SUCCESS_TTL_S` | `3600` | Cache TTL for successful search/scrape results. |
| `WEBRESEARCH_CACHE_SIZE` | `512` | LRU cache size (failures cached 60 s regardless). |
| `WEBRESEARCH_CRAWL_PAGE_DELAY_S` | `3` | Inter-page delay inside crawls (kept small so multi-page crawls stay fast). |
| `WEBRESEARCH_CONSENT_AUTOACCEPT` | `true` | Auto-click cookie-consent "accept" banners (Yahoo guce, OneTrust, Cookiebot). Set `false` to never click. |
| `TRADINGAGENTS_CAMOUFOX_HEADLESS` | `true` | Headless browser (set `false` to watch it work). |
| `TRADINGAGENTS_WEB_SCRAPE_TIMEOUT_S` | `120` | Per-page load timeout (s). |

Notes: crawl state is in-memory (a restart loses in-flight crawls); crawls
hold the single browser, so standalone scrapes queue behind an active crawl.

## Development

```bash
pip install -e ".[mcp,test,lint]"
ruff check .
pytest -m "unit or smoke"          # hermetic (CI runs exactly this)
pytest -m integration              # live network + Docker (auto-skip if missing)
```

Layered markers: `unit` (pure helpers, patched seams), `smoke` (real MCP
protocol over stdio/HTTP against a hermetic fake upstream), `integration`
(live web + real browser + Docker image).

## Responsible use

This is a research/reading tool for **publicly available** content, intended
for personal and internal use.

- Respect each site's **Terms of Service** and **`robots.txt`**. If a page or
  site blocks you, stop — don't keep trying to get past it.
- Don't use it to access **paywalled or protected** content, or to harvest
  content or personal data at scale.
- You are responsible for how you use it: scraping may be restricted by the
  site you're reading and by the law in your jurisdiction.

## License

Apache-2.0. Portions derived from TradingAgents (Apache-2.0) — see `NOTICE`.