searxng-crawl4ai-mcp

A self-hosted MCP server that gives an AI agent web search and page fetching, without depending on any commercial search or scraping API.

It is deliberately a thin layer. SearXNG and Crawl4AI are run as their official container images and are spoken to over their documented HTTP APIs. This repository contains no wrapper around their internals, which is what makes it possible to follow their releases instead of drifting away from them.

Why this exists

The obvious way to build this is to import the scraping library and call it directly. That road ends badly, and predictably: every upstream release changes an internal API, the wrapper breaks, and nothing notices until a search quietly returns nothing.

So the constraint here is stated up front and enforced by tests:

• No code calls upstream internals. Only documented HTTP endpoints.

• Upstream versions are declared in exactly one file, versions.env. Moving to a new release means editing a tag there and nothing else.

• Every version change is verified before it reaches a running host, by starting the real upstream containers and exercising all tools against them.

Related MCP server: websearch-mcp

Architecture

MCP client
   |  Streamable HTTP + bearer token
   v
reverse proxy  ->  mcp        this repository; the only code here
                    |
                    +-- HTTP -> searxng     official image, unmodified
                    +-- HTTP -> crawl4ai    official image, unmodified

Three containers, no database. The server holds no state: crawl job state lives in Crawl4AI.

Tools

Failures are returned, not thrown, and carry a machine-readable reason so the caller can tell "the site is down" apart from "that target is not allowed".

A search reports whether it actually returned the number of results it was asked for, and when it did not, why: the results ran out, the page limit was reached, the time budget was spent, or an upstream failed partway. An agent that gets fewer results than it asked for can otherwise not tell "the web has no more of this" from "this server stopped looking", and those call for opposite next moves.

A request to SearXNG or Crawl4AI that fails quickly - a refused connection, a 502, a rate limit - is retried with backoff. One that fails by using up its own timeout is not: repeating it would double a wait that has already proved too long. The rule is a single budget rather than a list of special cases, so retrying never adds more than two seconds to any call. web_crawl also reports why it stopped - a page limit, a depth limit or nothing left to visit - so a truncated crawl is visible instead of looking complete. Responses are capped at 25,000 characters and say so when they were cut.

Two ways to run it

Both entries are built from the same server factory, so neither can grow a capability the other lacks. MCP_TRANSPORT picks between them and defaults to http; the deployed stack is unaffected by the existence of the other one.

stdio exists so that trying this out does not require issuing a token and putting a reverse proxy in front of it. It is for one person on one machine:

MCP_TRANSPORT=stdio \
SEARXNG_URL=http://127.0.0.1:8081 \
CRAWL4AI_URL=http://127.0.0.1:11235 \
CRAWL4AI_API_TOKEN=... \
node dist/index.js

The outbound address policy, the fetch budget and the response size cap apply identically in both. What stdio drops is only what a listener needed.

Requirements

• Docker and Docker Compose

• Node.js 22 or newer, and pnpm, if you intend to work on the server itself

Getting started

cp .env.example .env
# fill in MCP_AUTH_TOKEN, MCP_ALLOWED_HOSTS and SEARXNG_SECRET

docker compose --env-file versions.env --env-file .env -f docker/compose.yaml up -d

.env is gitignored and must stay that way. Every environment-specific value lives there or in the deployment inventory, never in a tracked file.

Following upstream releases

1. A scheduled job notices a new SearXNG or Crawl4AI release and opens a pull request that changes only versions.env.

2. CI starts the whole stack on that version and runs every tool against it.

3. If it passes, a human decides whether to deploy. Nothing is deployed automatically.

To roll back, restore the previous versions.env and redeploy. Image tags are pinned, so the previous state is reproducible.

When the contract tests fail after a bump

Several behaviours here exist only to work around something the upstream does not do: the level-by-level crawl (Crawl4AI refuses a deep-crawl strategy from any HTTP caller by design), the paging and de-duplication of search results, the message-matching error classification, the override of Crawl4AI's short-page block verdict, and a few more. Each one is pinned by a test in tests/contract/tier-a/upstream.test.ts that fails the day its cause goes away, and the test name says which guarantee was withdrawn.

So a red test after a version bump is information, not only breakage: it may mean a workaround can now be removed. Read the failing test's name and the comment on the code it guards before changing anything, and do not redo the inventory by hand.

One check needs a model credential and is skipped, by name, wherever GEMINI_API_KEY is not set - which includes CI. Run the contract tests once on a machine that has the key before deploying a bump.

Outbound request policy

Fetch targets are resolved to IP addresses before the request is made, and private, loopback, link-local and cloud metadata ranges are refused. Additional ranges can be allowed through configuration. A refusal is reported distinctly from an unreachable host, so a blocked target is never mistaken for a broken one.

Because a name can resolve differently after it has been checked, the application-level check is a convenience that produces a clear error, not the security boundary. The boundary is a packet filter applied on the host during deployment.

Deployment

ansible/ deploys the stack to a single always-on Linux host. The playbook is idempotent and pulls prebuilt images; it never builds on the target, which matters when that target is a low-power machine.

Logged-in fetching (optional)

Some pages show their real content only to a signed-in member. Instead of storing site credentials in configuration, crawl4ai can run its browser on one saved profile: you sign in to a site once, by hand, in a Google Chrome window that the host opens on that profile, and every later fetch carries that login. Any number of sites can be added this way without a configuration change. A fetch that a site redirects to its login page fails with loginRequired, so an expired login is visible instead of silently returning the public page.

Enabling it

Set these in the host's host_vars (see ansible/inventory/host_vars.example.md); leaving mcp_login_session_root empty keeps the feature off.

• mcp_login_session_mountpoint - the disk that holds the store.

• mcp_login_session_root - the store itself, under the mount point. It holds profile/ (the browser profile) and state/.

• mcp_login_user - the host user that runs the login browser; the user whose desktop shows it. Required when mcp_login_session_root is set.

• mcp_login_display (:1) and mcp_login_browser (/usr/bin/google-chrome)

  • the X display to open the browser on, and the browser. The defaults fit a host with a single desktop on :1.

The host must already have Google Chrome and a way to show a browser on that display - a VNC desktop, a monitor, whichever it has. login-session only starts the browser there; it never starts or stops the display. This repository does not install or change either of them. The Ansible role creates the store only after checking that the disk is mounted, so an unmounted disk is never mistaken for an empty store. The store's root is mode 0711, so the login user can reach profile/ without being able to list the store.

Signing in

sudo <deploy dir>/login-session start    # pauses fetching, opens the login browser
# open the host's desktop and sign in (below)
sudo <deploy dir>/login-session stop     # saves the login, resumes fetching
sudo <deploy dir>/login-session status   # shows the marker, the login browser,
                                         # crawl4ai and the profile owner

For example, on a host whose desktop is a VNC server listening on a unix socket:

1. ssh <host> sudo /opt/searxng-crawl4ai-mcp/login-session start

2. Open the host's desktop on display :1 by whatever means the host offers. Today that is its VNC desktop: keep a tunnel open with ssh -N -L 5901:/run/vnc-desktop/vnc.sock <login user>@<host>

3. ...and connect a VNC client to localhost:5901. There is no VNC password; SSH is the gate.

4. Sign in in the Chrome window that start opened (a blank tab). The desktop's autostart also opens a Chrome on the default profile; do not confuse the two. Open chrome://version and use the window whose "Profile Path" is <store>/profile/Default.

5. ssh <host> sudo /opt/searxng-crawl4ai-mcp/login-session stop. It prints Login saved. Fetching has resumed.; the VNC client and the tunnel can then be closed.

start refuses to run when the host's Chrome and the crawler's Chromium have different major versions, because both write to the same profile and a newer major version can change its on-disk format. It prints both versions. Either update CRAWL4AI_SESSION_IMAGE to a build with the same major version as the host's Chrome, or hold the host's Chrome until the crawler catches up (sudo apt-mark hold google-chrome-stable).

start stops crawl4ai first because a Chromium profile can be opened by only one browser at a time. On each site, tick "keep me signed in" (or its equivalent): without it many sites issue a session cookie with no expiry, which the browser does not save when it closes, so the login would not survive stop.

To remove one site's login and keep the others, run start, open chrome://settings/content/all in the login browser, delete that site's data (or sign out on the site), then run stop.

While a login is in progress

Between start and the end of stop, fetches fail with loginInProgress rather than looking like an outage. Do not deploy during that time: crawl4ai refuses to start while a login is in progress, so the deployment's up --wait cannot complete.

Limitations

• A site that shows its login form inside the member page, instead of redirecting to a login page, cannot be detected as logged out.

• web_extract uses the login but does not detect an expired one, because it does not return the final URL.

• Other clients that use crawl4ai directly through the host's port 11235 (such as mcp-ec) get the same login, and the browser_config they send is ignored.

• During a login, those direct clients cannot reach crawl4ai either.

• While pages are fetched, the profile is owned by uid/gid 999, which may be an unrelated host user (such as dnsmasq); that user can read the profile too (see Security).

• When apt upgrades the host's Google Chrome to a new major version, login-session start stops at the version check. Update CRAWL4AI_SESSION_IMAGE to match, or sudo apt-mark hold google-chrome-stable.

Security

• This feature opens no new network listener. The login browser appears on the host's existing VNC desktop, which listens only on a Unix socket that only its owner can open (mode 0600, no TCP). Reaching it requires SSH as that user, so SSH is the gate, and the VNC traffic stays inside the SSH tunnel.

• The login browser runs with --password-store=basic. The VNC session runs a keyring, and without this flag Chrome encrypts cookies with the keyring's key, which the crawler's Chromium does not have, so the crawler could not read the login. With it, cookies are encrypted with Chromium's fixed built-in key, so the profile is protected only by its file permissions. You can check which key was used: the first three bytes of encrypted_value in <store>/profile/Default/Cookies are v10 for the fixed key and v11 for a keyring key.

• While pages are fetched, the profile belongs to uid/gid 999 (the crawl4ai container's user). If uid 999 is an existing host user (such as dnsmasq), that user's processes can read the profile, cookies included. This risk is accepted: changing the container's uid would mean re-owning every directory the upstream image writes to.

• The login applies to every URL this server fetches. Anyone who can call the MCP endpoint can read pages as the signed-in member, so sign in only to accounts whose pages you are prepared to expose to those callers.

Recovery

• If the host reboots during a login, the login browser is gone, crawl4ai stays stopped and the marker stays at login, so fetches keep failing with loginInProgress. Run login-session stop to recover: it returns the profile to the crawler and starts crawl4ai. login-session status shows the state first.

• Rolling CRAWL4AI_SESSION_IMAGE back to an older build can leave a profile that the older Chromium cannot open, because a newer Chromium has already written to it. Empty profile/ and sign in again. Only upgrades are covered by the guarantee that logins survive an image update.

Connecting Claude Code

Two installers register this checkout with Claude Code as a plugin. Neither builds anything: the plugin only tells Claude Code where the endpoint is and which bearer token to present, then allows the eight tools without a prompt. Both read MCP_PUBLIC_ENDPOINT and MCP_PUBLIC_AUTH_TOKEN (falling back to MCP_AUTH_TOKEN) from the gitignored .env.

# Linux / macOS
./install_claude_plugin.sh

# Windows - needs PowerShell 7 (pwsh), not Windows PowerShell 5.1
pwsh -ExecutionPolicy Bypass -File .\install_claude_plugin.ps1

-ExecutionPolicy Bypass matters when the checkout sits on a network share or came in as a download, where RemoteSigned refuses to run it. Restart Claude Code afterwards.

Using with LLMs

All eight tools are exposed to any LLM client that understands MCP. The HTTP transport requires a bearer token; stdio is for local development and single-user setups.

LLM configuration for structured extraction

web_extract uses Google's Gemini API to pull structured fields from pages. To enable it:

1. Get a free Gemini API key at Google AI Studio

2. Add to .env:

   GEMINI_API_KEY=<your-key>

3. Optionally override the model (default: gemini-flash-lite-latest):

   GEMINI_MODEL=gemini/gemini-2.0-flash

Without a key, web_extract degrades gracefully and returns the page as markdown. The other seven tools are unaffected and do not need LLM credentials.

Tool selection guide

Question answering: Use web_search_and_scrape to search and read top results in one call. This is faster and more accurate than searching alone, since you get the full source pages with the results.

Exploring a site: Start with web_map to list links, then web_scrape or web_crawl to read. This avoids fetching pages you don't need.

Single page: web_scrape reads one page as markdown, rendering JavaScript. Use it for dynamic content that plain HTTP fetch cannot read.

Multiple pages at once: web_batch_scrape fetches a list of URLs in parallel. Unlike calling web_scrape repeatedly, a failure on one URL does not stop the others, and you get the reason for each failure.

Crawling a site: web_crawl follows links from a starting URL up to a depth and page limit. It stays on the same host by default. When a crawl hits a limit, it reports which one, so you know whether to raise the limit or stop.

Extracting structured data: web_extract pulls specific fields out of a page (e.g. "product name, price, availability") in plain language. When LLM credentials are not configured, it returns the page as markdown and leaves the reading to you.

Long-running crawls: web_crawl returns immediately with a jobId. Use web_job_status to poll for results. The same id always reports the same state, so polling is safe.

Error handling

Every tool returns structured results with a status field: "ok", "failed", or "partial" (some URLs succeeded, others did not). When status is "failed", a failure field describes why:

• "access_denied" — the target is on a private, loopback, link-local or cloud metadata range

• "host_unreachable" — DNS failed, the connection timed out, or the server is not listening

• "http_error" — the server responded with a 4xx or 5xx status

• "parsing_failed" — the content exists but could not be parsed as HTML or markdown

• "response_size_exceeded" — the response was larger than 25,000 characters (truncated in output)

Distinguish between these when deciding what to do next. "Access denied" means the target is blocked by policy; "host unreachable" means trying again later might work; "http_error" on a 404 means the page does not exist.

Rate limiting and budgets

• Each tool call has a timeout (typically 30 seconds). Requests to SearXNG or Crawl4AI that fail quickly are retried with backoff, but those that exhaust their timeout are not, because repeating them would double the wait.

• Large crawls may return fewer pages than requested if they hit the page limit, depth limit, or time budget. The response reports which limit was reached.

• Responses are capped at 25,000 characters. When truncated, the output says so and you can web_job_status to retrieve the full results (for crawls) or re-fetch with different parameters.

Development

pnpm install
pnpm typecheck
pnpm lint
pnpm test

Contract tests come in two tiers. Tier A runs against a fixture site inside CI and gates merges. Tier B talks to the live internet, and is reported but not gating, because a datacenter IP being blocked by a search engine says nothing about whether this code is correct.

License

MIT