webresearcher-mcp-server

A light, standalone web-research MCP server: DuckDuckGo search, headless Camoufox page scraping (a hardened open-source Firefox), and same-domain crawling — zero Firecrawl, no API keys, no per-client browser download (in Docker mode).

It exposes four LLM-facing tools over MCP; the host LLM does the synthesis:

Failures are graceful by design: a blocked page or outage returns a short SCRAPE UNAVAILABLE / SEARCH UNAVAILABLE note (never an exception), and the note tells the LLM not to retry the same URL/query.

Why not just WebFetch / WebSearch?

This is not a replacement for your IDE's built-in fetch/search — it is the tool for JS-rendered, consent-gated, and multi-page research:

• web_scrape reads pages a bare HTTP fetch can't. It runs a real browser (Camoufox), so JavaScript renders and cookie-consent overlays are handled — where the built-in fetch returns an empty shell, this returns the content. When a page is protected or unreachable it degrades to a clear "rely on search" note instead of failing.

• web_crawl has no built-in equivalent (multi-page BFS + per-page blocked-page detection).

• web_search is free and independent of the harness's search availability; built-in search has the better generic index — use both.

• Plain public static pages: the built-in fetch is fine, no need to route everything through this server.

A typical session: web_search("BOX stock news") → web_scrape the 2–4 best results → optional web_crawl of the newsroom → synthesize with citations.

Related MCP server: Web Research MCP Server

Quick start (uvx — no Docker)

Run it straight from PyPI. uvx auto-installs the package (and its mcp extra) into an isolated env and runs the stdio server as an ordinary child process — no Docker, no install step, and it's cleaned up when your session ends:

uvx "webresearcher-mcp-server[mcp]" --help    # sanity check
uvx "webresearcher-mcp-server[mcp]"          # stdio server (default transport)

The Camoufox browser (used by web_scrape/web_crawl) is downloaded on first use (~5 min one-time) or pre-fetched with python -m camoufox fetch.

Shared HTTP server (optional — one detached process shared by every IDE/session so pacing + cache are global; the only Docker path we still use):

docker build -t webresearch-mcp .     # or: docker compose up -d
docker run -d --name webresearch-mcp -p 8001:8001 webresearch-mcp \
  --transport streamable-http --host 0.0.0.0 --port 8001

(healthcheck: curl -fs -X POST http://localhost:8001/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' → lists the four tools)

Integrating with Claude Code / Cursor

Claude Code

uvx (recommended — no Docker): add to your project .mcp.json (or claude mcp add). First use auto-installs the package into an isolated env (cached afterward); the server runs as an ordinary child process and is cleaned up when the session ends:

{ "mcpServers": {
    "webresearch": {
      "command": "uvx",
      "args": ["webresearcher-mcp-server[mcp]", "--transport", "stdio"]
    } } }

Avoid the docker run -i --rm per-session shape. It leaks orphaned containers: Claude Code closes the child's stdin pipe on exit, the container (owned by the Docker daemon) never receives a stop signal, so --rm never fires and stray containers pile up. Known issue: anthropics/claude-code#29058.

Docker-HTTP (one detached container shared by all sessions): run the shared container above, then:

{ "mcpServers": { "webresearch": { "type": "http",
                                    "url": "http://localhost:8001/mcp" } } }

Plain venv (no uv/uvx on the box): install the package in a venv, then:

{ "mcpServers": { "webresearch": {
      "command": "/path/to/.venv/bin/python",
      "args": ["-m", "webresearch_mcp"] } } }

Cursor

Same shapes in ~/.cursor/mcp.json (user) or .cursor/mcp.json (project). uvx (no Docker):

{ "mcpServers": { "webresearch": {
      "type": "stdio",
      "command": "uvx",
      "args": ["webresearcher-mcp-server[mcp]", "--transport", "stdio"] } } }

or the {"url": "http://localhost:8001/mcp"} form for the shared HTTP container. See docs/integration.md for the full walkthrough, including the optional cross-IDE plugin (plugin/): one directory, two manifests (Claude Code + Cursor), bundling the stdio server + an always-on research rule + a /research command + a web-researcher subagent — no Docker needed.

Plain Python (no uv/uvx)

python3 -m venv .venv && . .venv/bin/activate
pip install "webresearcher-mcp-server[mcp]"   # or: pip install -e ".[mcp]" from a clone
python -m camoufox fetch          # one-time browser download
webresearcher-mcp-server                   # stdio server
webresearcher-mcp-server --transport streamable-http --port 8001

Configuration (env vars)

Notes: crawl state is in-memory (a restart loses in-flight crawls); crawls hold the single browser, so standalone scrapes queue behind an active crawl.

Development

pip install -e ".[mcp,test,lint]"
ruff check .
pytest -m "unit or smoke"          # hermetic (CI runs exactly this)
pytest -m integration              # live network + Docker (auto-skip if missing)

Layered markers: unit (pure helpers, patched seams), smoke (real MCP protocol over stdio/HTTP against a hermetic fake upstream), integration (live web + real browser + Docker image).

Responsible use

This is a research/reading tool for publicly available content, intended for personal and internal use.

• Respect each site's Terms of Service and robots.txt. If a page or site blocks you, stop — don't keep trying to get past it.

• Don't use it to access paywalled or protected content, or to harvest content or personal data at scale.

• You are responsible for how you use it: scraping may be restricted by the site you're reading and by the law in your jurisdiction.

License

Apache-2.0. Portions derived from TradingAgents (Apache-2.0) — see NOTICE.