searxNcrawl
This MCP server gives AI agents web search plus clean Markdown/JSON page and site crawling on your own hardware.
Search the web via configured SearXNG and/or You.com providers in parallel, with merged/deduplicated results, provider attribution, and filters for language, time range, safe search, paging, engines/categories, and provider subset.
Crawl one or many URLs into Markdown or JSON, with timeouts, concurrency, exact/off deduplication, optional link removal, and Playwright
storage_statefor authenticated pages.Crawl an entire site from a seed URL using DFS within depth, page, timeout, and subdomain limits, returning Markdown or JSON.
Optionally include page content in search by crawling each result and adding its Markdown or
crawl_error.Use it via MCP tools (
search,crawl,crawl_site) or through the project's CLI/Python API.
Integrates with SearXNG to provide privacy-respecting web search capabilities, allowing users to search the web with configurable language, time range, categories, engines, and safe search.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@searxNcrawlsearch for latest AI trends"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
searxNcrawl is an open-source MCP server and CLI toolkit that gives AI agents web search and clean Markdown crawling, built on Crawl4AI. Search queries the configured providers, SearXNG and/or You.com, in parallel and merges the results. It runs on your own hardware, needs no API key with SearXNG and succeeds searxng-mcp.
SEARXNG_URL=http://your-searxng:8888 uvx searxncrawlQuick start
Connect your MCP client directly to searxNcrawl over STDIO or its HTTP endpoint.
You need uv and, for the search tool, at least one search provider: a SearXNG instance with JSON output enabled and/or a You.com API key (YDC_API_KEY). No SearXNG yet? Start one with Docker. Crawling works without either.
Add it to your agent
One step for every agent on your machine
uvx searxncrawl setupsetup detects Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop, asks for your SearXNG URL and, optionally, a You.com API key (hidden input, never printed), shows each config change and applies it only after you confirm. Every file it touches is backed up first, and setup --remove undoes it. More options are in MCP server.
Or per agent
The buttons and commands below assume SearXNG at http://localhost:8888; change SEARXNG_URL if yours runs elsewhere, or add YDC_API_KEY for You.com.
Claude Code
claude mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawlCodex
codex mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawlVS Code
code --add-mcp '{"name":"searxncrawl","command":"uvx","args":["searxncrawl"],"env":{"SEARXNG_URL":"http://localhost:8888"}}'Cursor (~/.cursor/mcp.json), Claude Desktop and other clients that use the mcpServers format
{
"mcpServers": {
"searxncrawl": {
"command": "uvx",
"args": ["searxncrawl"],
"env": { "SEARXNG_URL": "http://localhost:8888" }
}
}
}The first crawl downloads Playwright's Chromium once (about 550 MB). For Docker, HTTP transport and other setups see Installation and MCP server.
Claude Desktop bundle: download
searxncrawl-<version>.mcpbfrom GitHub Releases and open it in Claude Desktop.Smithery users: use the direct installation above or the native bundle; see the Smithery transition.
Ask your agent
Search for the Crawl4AI quick start, crawl the official page and summarise
how to run a first crawl.Your agent calls search, picks a result and calls crawl. It gets back clean Markdown instead of raw HTML.
Use the terminal
uvx --from searxncrawl crawl https://modelcontextprotocol.io/docs/getting-started/intro
SEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "model context protocol" --max-results 5
SEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "Python documentation" --max-results 3 --include-contentChoose how to connect
Mode | Setup | Hosting |
Native MCP over STDIO (default) |
| Runs alongside your MCP client |
Native MCP over HTTP |
| You operate the server; see HTTP configuration |
Arcade gateway (optional) | Separate Arcade adapter | Requires your own Playwright-compatible worker host and authenticated connection to Arcade |
Standard installations and releases do not require Arcade.
For shared HTTP access, use an authenticated TLS proxy. Enable Host/Origin checks with
FASTMCP_HTTP_HOST_ORIGIN_PROTECTION=truebefore startup; allowlists alone do not enable checks or authenticate callers.
Related MCP server: evo-scry
How it works
flowchart LR
agent["AI agent<br/>Claude Code · Codex<br/>Cursor · VS Code"]
server["searxNcrawl<br/>MCP server"]
searxng["SearXNG (your instance)<br/>and/or You.com API"]
browser["Crawl4AI +<br/>Playwright Chromium"]
results["Merged results<br/>attributed per provider"]
markdown["Deduplicated<br/>Markdown or JSON"]
agent -- "MCP · STDIO or HTTP" --> server
server -- "search" --> searxng --> results
server -- "crawl · crawl_site" --> browser --> markdownResults return to the agent as the tool response. Search and crawling are also available as the search and crawl CLI commands, crawling also as a Python API.
searxNcrawl exposes three MCP tools:
Tool | What it does |
| Queries the configured providers (SearXNG, You.com) in parallel and returns one merged, deduplicated list with the source of each result. Filters: language, time range, safe search, paging; categories and engines for SearXNG. With |
| Renders one or many URLs in Chromium and returns Markdown or JSON with metadata and references. |
| Follows links from a seed URL within depth, page and timeout limits, tuned for documentation sites. |
Both crawl tools drop sections that repeat within a page's Markdown (dedup_mode="exact"), can strip links for a smaller context (remove_links), and accept a Playwright storage_state for pages behind a login. Full parameters are in the MCP tools reference.
Private SearXNG instances
SEARXNG_EXTRA_HEADERSadds headers to SearXNG requests in both CLI and MCP, including access-gateway credentials and a custom User-Agent.Values such as
@~/.config/searxncrawl/cf_secretload secrets from files; absolute and~/paths are supported.Invalid header configuration disables SearXNG with a value-free warning; a configured You.com provider remains available.
Setup and JSON examples: SearXNG configuration.
Why searxNcrawl
Hosted search and crawl APIs are quick to start with, but every call goes through someone else's service and is billed per request or credit. searxNcrawl runs on your own machine or server.
searxNcrawl | ||||
Search and crawl run on your hardware | ✅ Python package or Docker; You.com queries go to You.com | ✅ Self-host stack (Docker Compose), or hosted API | ❌ Hosted API | ❌ Hosted API |
API key | Not needed with SearXNG; You.com needs one | Hosted API: needed. Self-hosted: not by default | Needed | Optional, anonymous use is rate-limited |
Cost per call | None with SearXNG; You.com per its plans | Hosted: free monthly credits, then paid credits. Self-hosted: none | Free monthly credits, then paid credits | Free monthly credits, then paid per request |
Search backend | Your SearXNG (the engines you enable) and/or You.com | Hosted: Firecrawl's search. Self-hosted: SearXNG if configured, otherwise DuckDuckGo | Tavily's search API | Exa's search index |
Licence | MIT | AGPL-3.0 | MCP server MIT, API proprietary | MCP server MIT, API proprietary |
Self-hosted Firecrawl can search through SearXNG too; it runs as a multi-service stack (API, Playwright service, Redis, RabbitMQ, PostgreSQL and more), while searxNcrawl is a single Python process with Chromium, plus your SearXNG instance if you use one.
Where the others are a better fit: if you do not want to run a search provider and a browser yourself, a hosted API is less work, and Exa's semantic search index or Firecrawl's extraction features go beyond what searxNcrawl does today. Comparison based on each project's public repository and documentation in October 2026; corrections are welcome. Firecrawl is a trademark of its owner; searxNcrawl is not affiliated with it.
FAQ
Only for the search tool, and only if you do not use You.com (YDC_API_KEY). crawl and crawl_site work without any search provider. SearXNG serves HTML only by default, so JSON output must be enabled (search.formats in settings.yml). Public instances may not enable it and often rate-limit automated requests, so run your own. Installation has a Docker one-liner.
Yes. Pages are rendered in Playwright's Chromium through Crawl4AI before they are converted to Markdown, so client-side content is captured.
Yes. Capture a session once with crawl-capture, then pass the saved storage_state file to crawl or crawl_site. See Session capture.
Start with
--transport http(port 8000 by default) or Docker Compose (port 9555).Put shared access behind an authenticated TLS proxy and connect clients to its
/mcpendpoint.Enable header protection and configure the externally visible Host and Origin allowlists.
The first crawl downloads Playwright's Chromium (about 550 MB on disk). Later runs reuse it. Set PLAYWRIGHT_AUTO_INSTALL=false where browsers are provisioned separately.
searxng-mcp is deprecated in favour of searxNcrawl. It only searched SearXNG and could pick a random public instance; searxNcrawl adds page and site crawling but needs your own SEARXNG_URL (or a You.com key), as random public-instance selection is not included.
No. Connect directly over STDIO or HTTP for the standard setup.
The optional adapter connects a separately hosted worker to Arcade's authentication and gateway.
Arcade Cloud's current runner cannot install Playwright; the worker needs a compatible host.
Documentation
Guide | Contents |
uvx, Docker Compose, pip, uv, search providers and environment variables | |
STDIO and HTTP transports, client configuration, Host and CORS guards | |
Parameters for | |
| |
| |
Markdown and JSON structure, | |
Authenticated crawling with a saved browser session | |
| |
Shared authentication and routing through Arcade; separate deployment | |
Modules and features for contributors |
Contributing
Bug reports, feature ideas and pull requests are welcome. Read the contributing guide and open an issue first for larger changes. Coding agents find project conventions in AGENTS.md. Report vulnerabilities privately as described in the security policy.
License
MIT — © 2026 DDM – Das Digitale Momentum GmbH & Co KG
Available Tools
3 toolscrawlCrawlC
Crawl one or more URLs and extract content as markdown or JSON.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | List of URLs to crawl (required). Accepts a single URL or multiple URLs. | |
| timeout | No | Per-URL timeout in seconds (default: 15, must be >= 1) | |
| dedup_mode | No | Markdown dedup mode - "exact" (default) or "off" | exact |
| concurrency | No | Maximum concurrent crawls (default: 3) | |
| remove_links | No | Remove all links from the markdown output (default: false) | |
| output_format | No | 'markdown' (default) or 'json' - markdown: Clean concatenated markdown with URL headers and timestamps - json: Full JSON with metadata, references, and statistics | markdown |
| storage_state | No | Path to Playwright storage_state JSON for authenticated crawling |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden, and it does little beyond naming the output formats. It does not disclose network/HTTP effects, the need for storage_state to reach authenticated pages, per-URL failure behavior, or how partial failures surface, all of which matter for a 7-parameter crawling tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler. It is efficient, though arguably under-specified rather than genuinely concise, since key operational context is simply absent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 7 parameters, no annotations, and no output schema, the description should carry more weight than one sentence. It omits authentication via storage_state, concurrency/timeout implications, and expected return shape, leaving the agent to reconstruct behavior from the schema alone.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents timeout, dedup_mode, concurrency, remove_links, output_format and storage_state. The description adds nothing beyond restating the markdown/JSON choice, which the schema itself explains in more detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (crawl) and resource (URLs) plus the extraction outputs (markdown or JSON), so the core action is unambiguous. However, it never distinguishes itself from the sibling crawl_site, leaving the agent to infer whether this is a page-level or site-level crawler.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No when-to-use guidance, no prerequisites, and no mention of alternatives. The agent is not told to prefer crawl_site for whole-site traversal or search for discovery, so tool selection against its two siblings rests entirely on the name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_siteCrawl SiteC
Crawl an entire website starting from a seed URL using DFS strategy.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The seed URL to start crawling from | |
| timeout | No | Overall site crawl timeout in seconds (default: 120, must be >= 1) | |
| max_depth | No | Maximum depth to crawl (default: 2, 0 = seed page only) | |
| max_pages | No | Maximum number of pages to crawl (default: 25) | |
| dedup_mode | No | Markdown dedup mode - "exact" (default) or "off" | exact |
| remove_links | No | Remove all links from the markdown output (default: false) | |
| output_format | No | Output format - "markdown" (default) or "json" - markdown: Clean concatenated markdown with URL headers and timestamps - json: Full JSON with metadata, references, and crawl statistics | markdown |
| storage_state | No | Path to Playwright storage_state JSON for authenticated crawling | |
| include_subdomains | No | Whether to include subdomains in the crawl (default: false) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It discloses the DFS traversal strategy, which is useful, but says nothing about rate limiting, scope restrictions, long-running behavior against the 120s default timeout, or authentication requirements despite a storage_state parameter.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler. It is efficient, though the extreme brevity borders on under-specification for a nine-parameter crawling tool rather than reflecting purposeful conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex, potentially long-running, resource-intensive crawler with nine parameters, no annotations, and no output schema, one sentence is insufficient. It omits operational limits, output characteristics, and safety considerations an agent needs before invoking it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with every parameter (timeout, max_depth, max_pages, dedup_mode, remove_links, output_format, storage_state, include_subdomains) fully documented in the schema. The description adds no parameter-level meaning beyond that, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Crawl) and resource (an entire website) along with the starting point (seed URL) and traversal strategy (DFS). The word 'entire' implies a multi-page operation distinct from the sibling 'crawl', but the differentiation from 'crawl' and 'search' is never made explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to choose this tool over the 'crawl' or 'search' siblings, nor any mention of prerequisites such as robots.txt compliance, politeness, or when authenticated crawling is warranted. Usage must be entirely inferred from the phrase 'entire website'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
searchSearchA
Search the web via the configured search providers (SearXNG, You.com).
All configured providers are queried in parallel. Their results are merged (URL dedup, round-robin, results found by several providers first) and each result lists the providers that returned it.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | Search query string (required) | |
| pageno | No | Page number for results (minimum 1). Default: 1 | |
| engines | No | SearXNG only. Specific search engines to use. Default: None (all available) | |
| language | No | Language code for results (e.g., 'en', 'de', 'fr'). Default: 'en' | en |
| providers | No | Subset of configured providers to query ('searxng', 'youcom'). Default: None (all configured providers) | |
| categories | No | SearXNG only. Categories to search (e.g., ['general', 'images', 'news']). Default: None (all) | |
| safesearch | No | Safe search level - 0 (off), 1 (moderate), 2 (strict). Default: 1 | |
| time_range | No | Time range filter - 'day', 'week', 'month', or 'year'. Default: None (no filter) | |
| max_results | No | Maximum results to return (1-50). Default: 10 | |
| max_retries | No | Maximum attempts per provider for transient request errors (default: 3) | |
| include_content | No | Also crawl each result page and add its Markdown as "markdown" (or "crawl_error" if the page fails). Default: false |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It helpfully discloses parallel provider querying, URL dedup, round-robin merging, and per-result provider attribution, but omits failure handling when one provider fails and rate-limit behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose is front-loaded in the first sentence, followed by concise behavioral detail. Three sentences with no wasted text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 11-parameter search tool with full schema coverage but no output schema, the description covers the essential aggregation behavior and result attribution. It could say more about result shape and partial-failure behavior, but it is adequate for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all 11 parameters. The description adds context about provider merging but no parameter-specific semantics beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb+resource ("Search the web") and names the configured providers. Siblings crawl and crawl_site are page-fetching tools, so the query-based discovery purpose is clearly distinguishable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by "Search the web", but there is no explicit guidance on when to use this versus crawl/crawl_site, nor any exclusions. The agent must infer that search is for discovery and crawl for known URLs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
v0.33.1- Changed
search1 field changed- added
Input schema / properties / include_contentAdded value: +{ + "default": false, + "description": "Also crawl each result page and add its Markdown as\n\"markdown\" (or \"crawl_error\" if the page fails). Default: false", + "type": "boolean" +}
1 tool update
v0.32.0- Changed
search4 fields changed- changed
Input schema / properties / categories / descriptionPrevious value: -"Categories to search (e.g., ['general', 'images', 'news']). Default: None (all)"New value: +"SearXNG only. Categories to search (e.g., ['general', 'images', 'news']). Default: None (all)" - changed
Input schema / properties / engines / descriptionPrevious value: -"Specific search engines to use. Default: None (all available)"New value: +"SearXNG only. Specific search engines to use. Default: None (all available)" - changed
Input schema / properties / max_retries / descriptionPrevious value: -"Maximum attempts for transient RequestError failures (default: 3)"New value: +"Maximum attempts per provider for transient request errors (default: 3)" - added
Input schema / properties / providersAdded value: +{ + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Subset of configured providers to query ('searxng', 'youcom').\nDefault: None (all configured providers)" +}
3 tool updates
v0.30.0- First observed
crawl - First observed
crawl_site - First observed
search
TDQS
Scored across 3 tools
The three tools have mostly distinct roles: search queries providers, crawl fetches specific URLs, and crawl_site traverses a whole site. The crawl vs. crawl_site boundary is clarified by the DFS/seed-URL description, though some overlap remains because crawl accepts multiple URLs.
All names are lowercase snake_case and action-oriented, which is readable and predictable. However, search and crawl are bare verbs while crawl_site uses a verb_noun compound, so the convention is not perfectly uniform.
Three tools are well scoped for a search-and-crawl server: one search entry point, one targeted URL crawler, and one whole-site crawler. Each tool earns its place without obvious redundancy.
The surface covers the core domain lifecycle: searching configured providers, extracting content from specific URLs, and crawling an entire site from a seed URL. No obvious missing operation prevents agents from accomplishing search-and-crawl workflows.
Maintenance
Related MCP Connectors
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Scrape, crawl and search the web for AI agents via MCP.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceMCP server for web search and content extraction using DuckDuckGo or SearXNG, with Playwright-based fetching and LLM-powered data extraction.139MIT
- AlicenseNot gradedqualityDmaintenanceMCP server for internet search via direct Google and DuckDuckGo HTML scraping with AI-powered result normalization and optional summarization, requiring no API keys for search.MIT
- AlicenseAqualityDmaintenanceA privacy-focused web search and content extraction MCP server. It integrates SearxNG with fallback to Google scraping, featuring relevance ranking, security-aware search, and rate limiting.3MIT
- AlicenseAqualityCmaintenanceMCP server for local web search via SearXNG, providing unlimited queries without API keys or cost, with automatic fallback to public instances.31MIT