Crawly-MCP

Browser-backed web search and page fetch for local LLMs, exposed as MCP tools and a CLI.

The design history is tracked in docs/IMPLEMENTATION_PLAN.md.

License

This project is licensed under the MIT License.

Naming

• Python distribution: crawly-mcp

• Import package: crawly_mcp

• CLI executable: crawly-cli

• MCP server executable: crawly-mcp

Tools

• search(provider, context) runs a browser-backed search on duckduckgo (default), google, or yandex and returns up to 5 organic result URLs. The opt-in searxng provider routes the query through a JSON-API call to a SearXNG instance you supply via CRAWLY_SEARXNG_URL; see below.

• fetch(urls, content_format) fetches 1..5 URLs and returns browser-rendered page content with per-URL pages, errors, and truncated fields. Use content_format="html" for raw HTML or content_format="text" for extracted readable text.

• page_search(url, query) searches for content on a single page. Tries known site-search facilities first (Algolia DocSearch, OpenSearch descriptor, Readthedocs API), then generic GET form detection, then find-in-page text as a fallback. Returns a mode discriminator plus up to 5 results with snippets and optional result URLs.

context is intentionally the search query string for caller compatibility.

Setup

uv sync
chromium --version

For host usage, crawly defaults to launching a system Chromium binary. If Chromium is installed in a non-standard location, set:

PLAYWRIGHT_CHROMIUM_EXECUTABLE=/path/to/chromium

To force Playwright-managed Chromium instead of a host browser:

PLAYWRIGHT_BROWSER_SOURCE=bundled

SearXNG (opt-in)

The searxng provider is opt-in. To use it, both conditions must hold:

1. Pass provider="searxng" to the search MCP tool (or --provider searxng on the CLI). It is not the default.

2. Set CRAWLY_SEARXNG_URL to the URL of a SearXNG instance whose JSON output is enabled (search.formats: [..., json] in its settings.yml).

Without CRAWLY_SEARXNG_URL, the provider returns an invalid_input error.

CRAWLY_SEARXNG_URL=http://127.0.0.1:8080/ \
  uv run crawly-cli search --provider searxng --context "..."

There is no instance registry and no automatic cross-provider fallback — failures from your SearXNG instance surface directly to the caller. This is deliberate: pinning is an explicit choice and crawly respects it.

Run alongside a local SearXNG (compose example)

A self-contained recipe is in examples/searxng-compose/: a docker-compose.yml that runs both searxng and crawly-mcp on a shared network, plus a settings.yml with search.formats: [html, json] and server.limiter: false (the two things crawly's searxng provider needs from any instance). The crawly image is pulled from GHCR by default; ports are controlled via a .env file.

cd examples/searxng-compose
cp .env.example .env        # edit if you want a non-default host port or local image
docker compose up -d
# MCP HTTP at http://127.0.0.1:10000/mcp/   (CRAWLY_HOST_PORT)

SearXNG itself is reachable only on the compose network (http://searxng:8080/ from crawly); no host port is mapped by default.

See examples/searxng-compose/README.md for the env-var reference and a smoke-test snippet.

Why public instances generally won't work

Public SearXNG instances listed on https://searx.space typically respond to programmatic clients with 429, an HTTP redirect to /, or a 200 with an empty results page — driven by SearXNG's built-in botdetection middleware. The community of instance operators actively discourages automated scraping (it burns down their upstream Google/Bing rate-limit budget). This is why the provider expects a SearXNG you control: self-hosted, on your LAN, or otherwise configured to permit JSON access for trusted clients.

Usage

Run the CLI directly:

uv run crawly-cli search --context "python async playwright"
uv run crawly-cli fetch https://example.com
uv run crawly-cli page-search --url https://docs.example.com/guide --query "authentication"

The page-search subcommand prints a JSON PageSearchResponse with mode, attempted, results_url, and results[].

Run the MCP server over stdio:

uv run crawly-mcp

Expose HTTP transport instead of stdio:

uv run crawly-mcp --transport streamable-http --host 127.0.0.1 --port 8000

The MCP server also reads:

• CRAWLY_HOST

• CRAWLY_PORT

Container

The container image uses Patchright-managed Chromium on a plain Python Debian base and defaults to HTTP MCP on port 8000.

Build locally:

docker build -t crawly-mcp:local .

Run locally:

docker run --rm --init -p 8000:8000 crawly-mcp:local

Override the transport to stdio:

docker run --rm --init -i crawly-mcp:local crawly-mcp --transport stdio

Launch the stdio MCP server from the current checkout with an auto-build step:

./scripts/run_crawly_mcp_stdio_container.sh

Launch the HTTP MCP server from the current checkout:

./scripts/run_crawly_mcp_http_container.sh

The container defaults to:

• PLAYWRIGHT_BROWSER_SOURCE=bundled

• CRAWLY_HOST=0.0.0.0

• CRAWLY_PORT=8000

• CRAWLY_FETCH_MAX_SIZE=1048576

• CRAWLY_PROFILE_DIR=/data/profiles

• CRAWLY_PROFILE_CLEANUP_ON_START=true

For local LLMs with smaller context windows, call fetch(..., content_format="text") and lower the payload cap:

CRAWLY_FETCH_MAX_SIZE=16384 ./scripts/run_crawly_mcp_stdio_container.sh

The HTTP MCP endpoint is unauthenticated in v1. Deploy it behind localhost, a private network, or an auth/TLS reverse proxy.

Published images are intended to be:

• ghcr.io/<owner>/crawly-mcp

• <dockerhub-namespace>/crawly-mcp

The first GHCR publish may need a one-time manual visibility change to make the package public.

Integration Setup

Docker Run

Run the published GHCR image directly:

docker run --rm --init \
  -p 8000:8000 \
  -e CRAWLY_HOST=0.0.0.0 \
  -e CRAWLY_PORT=8000 \
  -e CRAWLY_FETCH_MAX_SIZE=16384 \
  -e CRAWLY_BROWSER_LANG=en-US \
  -e CRAWLY_BROWSER_LOCATION=America/New_York \
  ghcr.io/dshein-alt/crawly-mcp:latest

The most important runtime overrides are:

• CRAWLY_FETCH_MAX_SIZE: caps returned fetch payload size for both content_format="html" and content_format="text".

• CRAWLY_BROWSER_LANG: sets browser locale and primary Accept-Language.

• CRAWLY_BROWSER_LOCATION: sets browser timezone / location persona.

MCP Client Config

For MCP clients that can launch a local command, point them at the project script so the server comes from the current checkout:

mcpServers:
  - name: Crawly MCP
    command: /path/to/crawly/scripts/run_crawly_mcp_stdio_container.sh
    args: []
    env:
      CRAWLY_CONTAINER_ENGINE: docker

Replace /path/to/crawly with your checkout path. The launcher rebuilds crawly-mcp:local before starting the stdio server so container contents stay aligned with local source changes. Set CRAWLY_MCP_SKIP_BUILD=1 if you want to skip that build when the local image is already current.

For clients that support HTTP MCP, start a local or published crawly-mcp HTTP server first, then point the client at the running instance:

http://127.0.0.1:8000/mcp

For Continue, an HTTP MCP config looks like:

name: New MCP server
version: 0.0.1
schema: v1
mcpServers:
  - name: Crawly
    type: streamable-http
    url: http://127.0.0.1:8000/mcp

The url must match an actually running crawly-mcp HTTP instance.

If your client's MCP config accepts direct URLs, the entry is typically shaped like:

mcpServers:
  - name: Crawly MCP
    url: http://127.0.0.1:8000/mcp

Set CRAWLY_HTTP_BIND_HOST or CRAWLY_HTTP_BIND_PORT before launching if you need the listener on a different interface or port.

Bundled Skill / Prompt

This repo includes two reusable instruction files for small-context web workflows:

• skills/web-search/SKILL.md — Codex skill guidance under the web-search skill name

• skills/continue-web-search.md — Continue-native invokable prompt named Web Search

Use them when a small local model must search, fetch, and synthesize across multiple pages without overflowing context.

Browser configuration

crawly uses patchright (a Playwright fork with bundled fingerprint patches) and keeps a small set of per-search-provider persistent profiles on disk. The following env vars tune the browser persona and search trace capture:

Profile persistence

Each provider (duckduckgo, google, yandex) keeps its own subdirectory under CRAWLY_PROFILE_DIR with cookies, localStorage, and session state. In Docker, mount a named volume at whatever path CRAWLY_PROFILE_DIR points to (default in the image: /data/profiles):

docker run -v crawly-profiles:/data/profiles crawly-mcp

Fingerprint canary

scripts/fingerprint_check.py runs a set of JS assertions against a blank page to verify the browser's JS-visible fingerprint looks like real Chrome:

uv run python scripts/fingerprint_check.py --verbose

Exits non-zero if any check fails. CI runs this on release tags.

Search tracing

Tracing is disabled by default. Set CRAWLY_TRACE_DIR only when you want to compare an automated run with manually collected artifacts:

CRAWLY_TRACE_DIR=./dump/trace uv run crawly-mcp --transport streamable-http

Each traced search() call writes one directory containing:

• meta.json with provider, query, warm-up/jitter data, final URL/title, and parsed result URLs

• fingerprint.json with JS-visible browser properties

• network.jsonl with request/response/failure events

• page.html and screenshot.png from the terminal page state

Design Notes

• One shared incognito browser per process for fetch() (fresh context per request). search() uses per-provider persistent contexts with on-disk profiles keyed by provider.

• PLAYWRIGHT_BROWSER_SOURCE=system uses a host Chromium binary (driven by patchright).

• PLAYWRIGHT_BROWSER_SOURCE=bundled uses patchright-managed Chromium (patchright install chromium).

• Global navigation concurrency cap of 3.

• Timeouts: 15s per page, 20s total for search, 35s total for fetch.

• SSRF guard: http/https only, no embedded credentials, blocks loopback/private/link-local/reserved IPs before navigation and on browser subrequests.

• JavaScript challenge pages get a bounded 10s settle window. patchright provides fingerprint patches against common bot-detection checks; provider-specific warm-up hops and synthetic client-hint headers keep the browser identity stable across requests. No CAPTCHA solving or site-specific bypass logic.

• fetch() returns raw HTML by default, or extracted readable text when the request sets content_format="text".

• Returned fetch content is capped at 1 MiB per URL by default; set CRAWLY_FETCH_MAX_SIZE lower when you need smaller MCP payloads. This applies to both content_format="html" and content_format="text". Oversized responses are truncated and reported in truncated.

• robots.txt is not consulted in v1.

Development

source .venv/bin/activate
ruff check .
pytest

Smoke checks:

rg -n "web-search|web_search_mcp" README.md AGENTS.md CHANGELOG.md pyproject.toml src tests
.venv/bin/python scripts/http_mcp_smoke.py --url http://127.0.0.1:8000/mcp

Parser tests run against saved HTML fixtures; selector drift is an expected maintenance cost.