web-search-mcp
Provides web search capabilities using DuckDuckGo as the search provider.
Provides web search capabilities using Google as the search provider.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Crawly-MCP
Browser-backed web search and page fetch for local LLMs, exposed as MCP tools and a CLI.
The design history is tracked in docs/IMPLEMENTATION_PLAN.md.
License
This project is licensed under the MIT License.
Related MCP server: Web Search MCP
Naming
Python distribution:
crawly-mcpImport package:
crawly_mcpCLI executable:
crawly-cliMCP server executable:
crawly-mcp
Tools
Every tool result carries status, next_action, and detail first, so a caller can tell "the web has nothing on this" apart from "this provider could not answer" without a human having to say "try google".
| what it means |
|
| usable data returned |
|
| the source answered and genuinely had nothing |
|
| CAPTCHA, consent wall, bot page |
|
| not configured, unreachable, interstitial, or the page shape is no longer parseable |
|
| the source did not answer in budget |
|
| the request failed at the network layer |
|
| the URL was refused by the safety guard |
|
| the arguments are wrong |
|
An empty urls list with a status other than no_results is a tool failure, never an empty web.
search(provider, context)runs a browser-backed search and cascades providers in the configured priority order until one answers.providerdefaults toauto(start from the configured order); naming one starts there and cascades from it. Returnsurlsplusprovider(the winner),attempted(one entry per provider tried, with its status, page kind, and final URL),available_providers, andsuggested_providers.fetch(urls, content_format)fetches1..5URLs. Each URL is handled on its own: a refused, unsafe, unreachable, timed-out, or challenge-walled URL lands inerrorswith its own reason andnext_action, and the other URLs still return. Partial success is normal, sostatusisokwhenever at least one URL returned content.content_format="html"returns raw HTML,"text"returns extracted readable text.page_search(url, query)searches within a single page: known site-search facilities first (Algolia DocSearch, OpenSearch descriptor, Readthedocs API), then GET form detection, then find-in-page text.attemptedreports one entry per tier asok,empty,failed, orunavailable, so a broken site-search is not reported as a site with nothing on it.providers()reports which providers this deployment can use, which are excluded by configuration or missing settings, the cascade order, and how each provider last behaved in this process. Call it before blaming a search failure on the query.
context is intentionally the search query string for caller compatibility.
What the cascade does, and does not, retry
A provider is retried past when it is blocked, unavailable, timeout, or navigation_failed. A provider that reports no_results stops the cascade: an honest empty answer is an answer, and cascading on top of it turns "the topic is empty" into three more requests. Providers that fail repeatedly are skipped as fill slots in later calls, and the providers tool reports them as skipped.
Automatic browser checks and incomplete page shells get up to three seconds to reach results, within the provider attempt budget. Human verification walls are reported as blocked immediately. If a page read stalls or navigation interrupts it, search preserves the last successful reading. The budget also reserves time for trace capture and page cleanup.
Configuring provider priority and exclusion
Operators own the provider set; callers only ever see what survives it.
# Prefer a self-hosted SearXNG, then DuckDuckGo; never touch Google or Yandex.
export CRAWLY_SEARCH_PROVIDER_ORDER="searxng,duckduckgo"
export CRAWLY_SEARCH_PROVIDER_EXCLUDE="google,yandex"
# One provider per call: no cascading, lowest latency, cheapest bill.
export CRAWLY_SEARCH_MAX_PROVIDERS=1A provider missing from
CRAWLY_SEARCH_PROVIDER_ORDERis not used.A provider in
CRAWLY_SEARCH_PROVIDER_EXCLUDEis never used, even when a caller names it — that call is refused asinvalid_inputsaying it was excluded, andproviders()lists it withexcluded: trueand the reason.providers()reports the resultingcascade_order, so the setting can be verified without guessing.
Setup
uv sync
chromium --versionFor host usage, crawly defaults to launching a system Chromium binary. If Chromium is installed in a non-standard location, set:
PLAYWRIGHT_CHROMIUM_EXECUTABLE=/path/to/chromiumTo force Playwright-managed Chromium instead of a host browser:
PLAYWRIGHT_BROWSER_SOURCE=bundledSearXNG
The searxng provider routes the query through a JSON-API call to a SearXNG instance you supply, and it is the only provider here that is not fighting a bot wall — so when configured, it leads the cascade order.
To use it:
Set
CRAWLY_SEARXNG_URLto the URL of a SearXNG instance whose JSON output is enabled (search.formats: [..., json]in itssettings.yml).Ask for it by name (
provider="searxng",--provider searxng) or leaveprovider="auto"and let the cascade reach it.
Without CRAWLY_SEARXNG_URL the provider is reported unavailable and is left out of the cascade; asking for it by name is refused as invalid_input.
CRAWLY_SEARXNG_URL=http://127.0.0.1:8080/ \
uv run crawly-cli search --provider searxng --context "..."There is no instance registry: public instances on https://searx.space generally return 429 / redirects / empty results to non-browser clients, so this provider is for self-hosted or privately-known instances. When a SearXNG instance refuses, the cascade moves on to the next provider in the order and reports the instance's outcome in attempted.
Run alongside a local SearXNG (compose example)
A self-contained recipe is in examples/searxng-compose/: a docker-compose.yml that runs both searxng and crawly-mcp on a shared network, plus a settings.yml with search.formats: [html, json] and server.limiter: false (the two things crawly's searxng provider needs from any instance). The crawly image is pulled from GHCR by default. Its .env.example exposes the image, host port, log level, provider order, provider exclusions, and per-search provider attempt limit.
cd examples/searxng-compose
cp .env.example .env # edit provider, image, port, or logging settings
docker compose up -d
# MCP HTTP at http://127.0.0.1:10000/mcp/ (CRAWLY_HOST_PORT)The Compose file already passes CRAWLY_SEARCH_PROVIDER_ORDER, CRAWLY_SEARCH_PROVIDER_EXCLUDE, and CRAWLY_SEARCH_MAX_PROVIDERS from .env into the container. After changing them on a running deployment, run docker compose up -d --force-recreate crawly-mcp; no image rebuild or Compose-file edit is needed. Call the MCP providers tool to verify the resulting cascade_order.
SearXNG itself is reachable only on the compose network (http://searxng:8080/ from crawly); no host port is mapped by default.
See examples/searxng-compose/README.md for the env-var reference, provider-configuration examples, and a smoke-test snippet.
Why public instances generally won't work
Public SearXNG instances listed on https://searx.space typically respond to programmatic clients with 429, an HTTP redirect to /, or a 200 with an empty results page — driven by SearXNG's built-in botdetection middleware. The community of instance operators actively discourages automated scraping (it burns down their upstream Google/Bing rate-limit budget). This is why the provider expects a SearXNG you control: self-hosted, on your LAN, or otherwise configured to permit JSON access for trusted clients.
Usage
Run the CLI directly:
uv run crawly-cli search --context "python async playwright"
uv run crawly-cli search --provider google --context "python"
uv run crawly-cli fetch https://example.com --content-format text
uv run crawly-cli page-search --url https://docs.example.com/guide --query "authentication"
uv run crawly-cli providersEach subcommand prints its response envelope as JSON: search prints status, next_action, detail, urls, provider, attempted[], available_providers, suggested_providers; page-search prints status, mode, attempted[] (per tier), results_url, and results[].
Run the MCP server over stdio:
uv run crawly-mcpExpose HTTP transport instead of stdio:
uv run crawly-mcp --transport streamable-http --host 127.0.0.1 --port 8000The MCP server also reads:
CRAWLY_HOSTCRAWLY_PORT
Container
The container image uses Patchright-managed Chromium on a plain Python Debian base and defaults to HTTP MCP on port 8000.
Build locally:
docker build -t crawly-mcp:local .Run locally:
docker run --rm --init -p 8000:8000 crawly-mcp:localOverride the transport to stdio:
docker run --rm --init -i crawly-mcp:local crawly-mcp --transport stdioLaunch the stdio MCP server from the current checkout with an auto-build step:
./scripts/run_crawly_mcp_stdio_container.shLaunch the HTTP MCP server from the current checkout:
./scripts/run_crawly_mcp_http_container.shThe container defaults to:
PLAYWRIGHT_BROWSER_SOURCE=bundledCRAWLY_HOST=0.0.0.0CRAWLY_PORT=8000CRAWLY_FETCH_MAX_SIZE=1048576CRAWLY_PROFILE_DIR=/data/profilesCRAWLY_PROFILE_CLEANUP_ON_START=true
For local LLMs with smaller context windows, call fetch(..., content_format="text") and lower the payload cap:
CRAWLY_FETCH_MAX_SIZE=16384 ./scripts/run_crawly_mcp_stdio_container.shThe HTTP MCP endpoint is unauthenticated in v1. Deploy it behind localhost, a private network, or an auth/TLS reverse proxy.
Published images are intended to be:
ghcr.io/<owner>/crawly-mcp<dockerhub-namespace>/crawly-mcp
The first GHCR publish may need a one-time manual visibility change to make the package public.
Integration Setup
Docker Run
Run the published GHCR image directly:
docker run --rm --init \
-p 8000:8000 \
-e CRAWLY_HOST=0.0.0.0 \
-e CRAWLY_PORT=8000 \
-e CRAWLY_FETCH_MAX_SIZE=16384 \
-e CRAWLY_BROWSER_LANG=en-US \
-e CRAWLY_BROWSER_LOCATION=America/New_York \
ghcr.io/dshein-alt/crawly-mcp:latestThe most important runtime overrides are:
CRAWLY_FETCH_MAX_SIZE: caps returned fetch payload size for bothcontent_format="html"andcontent_format="text".CRAWLY_BROWSER_LANG: sets browser locale and primaryAccept-Language.CRAWLY_BROWSER_LOCATION: sets browser timezone / location persona.
MCP Client Config
For MCP clients that can launch a local command, point them at the project script so the server comes from the current checkout:
mcpServers:
- name: Crawly MCP
command: /path/to/crawly/scripts/run_crawly_mcp_stdio_container.sh
args: []
env:
CRAWLY_CONTAINER_ENGINE: dockerReplace /path/to/crawly with your checkout path. The launcher rebuilds
crawly-mcp:local before starting the stdio server so container contents stay aligned
with local source changes. Set CRAWLY_MCP_SKIP_BUILD=1 if you want to skip that build
when the local image is already current.
For clients that support HTTP MCP, start a local or published crawly-mcp HTTP server first,
then point the client at the running instance:
http://127.0.0.1:8000/mcpFor Continue, an HTTP MCP config looks like:
name: New MCP server
version: 0.0.1
schema: v1
mcpServers:
- name: Crawly
type: streamable-http
url: http://127.0.0.1:8000/mcpThe url must match an actually running crawly-mcp HTTP instance.
If your client's MCP config accepts direct URLs, the entry is typically shaped like:
mcpServers:
- name: Crawly MCP
url: http://127.0.0.1:8000/mcpSet CRAWLY_HTTP_BIND_HOST or CRAWLY_HTTP_BIND_PORT before launching if you need the
listener on a different interface or port.
Bundled Skill / Prompt
This repo includes two reusable instruction files for small-context web workflows:
skills/web-search/SKILL.md — Codex skill guidance under the
web-searchskill nameskills/continue-web-search.md — Continue-native invokable prompt named
Web Search
Use them when a small local model must search, fetch, and synthesize across multiple pages without overflowing context.
Browser configuration
crawly uses patchright (a Playwright fork with bundled fingerprint patches) and keeps a small set of per-search-provider persistent profiles on disk. The following env vars tune the browser persona and search trace capture:
Env var | Default | Purpose |
|
| Browser |
|
| Browser timezone id. |
|
| Browser viewport in |
|
| Max bytes returned per fetched URL after rendering the configured content format. This limit applies to both raw HTML and extracted text. Lower it for local LLMs with small context windows. |
|
| Toggle per-provider persistent search profiles. Set to |
|
| Parent directory for per-provider persistent profiles. Must be a writable mount in containers. Ignored when |
|
| Enable age-based profile cleanup at startup. Set to |
|
| Age threshold for profile cleanup. |
|
| Min/max ms delay between warm-up and real query. Two-int CSV. |
|
| Provider priority order. The cascade walks it in this sequence; an explicitly requested provider is still tried first. Unknown names are ignored with a warning. |
| unset | Providers to exclude entirely. An excluded provider is never tried, even when requested by name (that is refused as |
|
| How many providers one |
| unset | Exact hostnames (CSV) allowed to resolve to non-public addresses, for private-network deployments that must reach an internal wiki, search instance, or fixture site. No CIDRs, no wildcards — every entry is a hole in the guard, so name them. Scheme and credential checks still apply. |
| unset | Opt-in per-search artifact dump directory. When set, each |
Profile persistence
Each provider (duckduckgo, google, yandex) keeps its own subdirectory under CRAWLY_PROFILE_DIR with cookies, localStorage, and session state. In Docker, mount a named volume at whatever path CRAWLY_PROFILE_DIR points to (default in the image: /data/profiles):
docker run -v crawly-profiles:/data/profiles crawly-mcpFingerprint canary
scripts/fingerprint_check.py runs a set of JS assertions against a blank page to verify the browser's JS-visible fingerprint looks like real Chrome:
uv run python scripts/fingerprint_check.py --verboseExits non-zero if any check fails. CI runs this on release tags.
Search tracing
Tracing is disabled by default. Set CRAWLY_TRACE_DIR only when you want to compare an automated run with manually collected artifacts:
CRAWLY_TRACE_DIR=./dump/trace uv run crawly-mcp --transport streamable-httpEach traced search() call writes one directory containing:
meta.jsonwith provider, query, warm-up/jitter data, final URL/title, and parsed result URLsfingerprint.jsonwith JS-visible browser propertiesnetwork.jsonlwith request/response/failure eventspage.htmlandscreenshot.pngfrom the terminal page state
Design Notes
One shared incognito browser per process for
fetch()(fresh context per request).search()uses per-provider persistent contexts with on-disk profiles keyed by provider.PLAYWRIGHT_BROWSER_SOURCE=systemuses a host Chromium binary (driven by patchright).PLAYWRIGHT_BROWSER_SOURCE=bundleduses patchright-managed Chromium (patchright install chromium).Global navigation concurrency cap of
3.Timeouts:
15sper provider attempt,45sacross asearchcascade;15sper URL and35stotal forfetch.SSRF guard:
http/httpsonly, no embedded credentials, blocks loopback/private/link-local/reserved IPs before navigation and on browser subrequests.JavaScript challenge pages get a bounded
10ssettle window.patchrightprovides fingerprint patches against common bot-detection checks; provider-specific warm-up hops and synthetic client-hint headers keep the browser identity stable across requests. No CAPTCHA solving or site-specific bypass logic.fetch()returns raw HTML by default, or extracted readable text when the request setscontent_format="text".Returned fetch content is capped at
1 MiBper URL by default; setCRAWLY_FETCH_MAX_SIZElower when you need smaller MCP payloads. This applies to bothcontent_format="html"andcontent_format="text". Oversized responses are truncated and reported intruncated.robots.txtis not consulted in v1.
Development
source .venv/bin/activate
ruff check .
pytestSmoke checks:
rg -n "web-search|web_search_mcp" README.md AGENTS.md CHANGELOG.md pyproject.toml src tests
.venv/bin/python scripts/http_mcp_smoke.py --url http://127.0.0.1:8000/mcpParser tests run against saved HTML fixtures; selector drift is an expected maintenance cost.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
A real browser for your agent: screenshots, page checks, Markdown and PDFs.
Hosted browser for agents: open pages, search the web, extract data, read PDFs and take screenshots.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Related MCP Servers
- AlicenseBqualityDmaintenanceEnables LLMs to interact with web pages, take screenshots, and execute JavaScript in a real browser environment10297MIT
- AlicenseNot gradedqualityDmaintenanceEnables web searching through Google, DuckDuckGo, and Bing using a headless Chrome browser, returning structured results with titles, URLs, and snippets. Also supports fetching and extracting text content from any webpage.16MIT
- AlicenseAqualityDmaintenanceProvides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.346MIT
- FlicenseNot gradedqualityDmaintenanceEnables AI agents to perform web searches, extract webpage content, and conduct end-to-end search-and-extract operations using multiple search providers and content extraction methods.-