Skip to main content
Glama

searxNcrawl is an open-source MCP server and CLI toolkit that gives AI agents web search and clean Markdown crawling, built on Crawl4AI. Search queries the configured providers, SearXNG and/or You.com, in parallel and merges the results. It runs on your own hardware, needs no API key with SearXNG and succeeds searxng-mcp.

SEARXNG_URL=http://your-searxng:8888 uvx searxncrawl

Quick start

Connect your MCP client directly to searxNcrawl over STDIO or its HTTP endpoint.

You need uv and, for the search tool, at least one search provider: a SearXNG instance with JSON output enabled and/or a You.com API key (YDC_API_KEY). No SearXNG yet? Start one with Docker. Crawling works without either.

Add it to your agent

One step for every agent on your machine

uvx searxncrawl setup

setup detects Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop, asks for your SearXNG URL and, optionally, a You.com API key (hidden input, never printed), shows each config change and applies it only after you confirm. Every file it touches is backed up first, and setup --remove undoes it. More options are in MCP server.

Or per agent

The buttons and commands below assume SearXNG at http://localhost:8888; change SEARXNG_URL if yours runs elsewhere, or add YDC_API_KEY for You.com.

Claude Code

claude mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl

Codex

codex mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl

VS Code

code --add-mcp '{"name":"searxncrawl","command":"uvx","args":["searxncrawl"],"env":{"SEARXNG_URL":"http://localhost:8888"}}'

Cursor (~/.cursor/mcp.json), Claude Desktop and other clients that use the mcpServers format

{
  "mcpServers": {
    "searxncrawl": {
      "command": "uvx",
      "args": ["searxncrawl"],
      "env": { "SEARXNG_URL": "http://localhost:8888" }
    }
  }
}

The first crawl downloads Playwright's Chromium once (about 550 MB). For Docker, HTTP transport and other setups see Installation and MCP server.

  • Claude Desktop bundle: download searxncrawl-<version>.mcpb from GitHub Releases and open it in Claude Desktop.

  • Smithery users: use the direct installation above or the native bundle; see the Smithery transition.

Ask your agent

Search for the Crawl4AI quick start, crawl the official page and summarise
how to run a first crawl.

Your agent calls search, picks a result and calls crawl. It gets back clean Markdown instead of raw HTML.

Use the terminal

uvx --from searxncrawl crawl https://modelcontextprotocol.io/docs/getting-started/intro
SEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "model context protocol" --max-results 5
SEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "Python documentation" --max-results 3 --include-content

Choose how to connect

Mode

Setup

Hosting

Native MCP over STDIO (default)

uvx searxncrawl or uvx searxncrawl setup

Runs alongside your MCP client

Native MCP over HTTP

uvx searxncrawl --transport http or Docker Compose

You operate the server; see HTTP configuration

Arcade gateway (optional)

Separate Arcade adapter

Requires your own Playwright-compatible worker host and authenticated connection to Arcade

  • Standard installations and releases do not require Arcade.

  • For shared HTTP access, use an authenticated TLS proxy. Enable Host/Origin checks with FASTMCP_HTTP_HOST_ORIGIN_PROTECTION=true before startup; allowlists alone do not enable checks or authenticate callers.

Related MCP server: evo-scry

How it works

flowchart LR
    agent["AI agent<br/>Claude Code · Codex<br/>Cursor · VS Code"]
    server["searxNcrawl<br/>MCP server"]
    searxng["SearXNG (your instance)<br/>and/or You.com API"]
    browser["Crawl4AI +<br/>Playwright Chromium"]
    results["Merged results<br/>attributed per provider"]
    markdown["Deduplicated<br/>Markdown or JSON"]

    agent -- "MCP · STDIO or HTTP" --> server
    server -- "search" --> searxng --> results
    server -- "crawl · crawl_site" --> browser --> markdown

Results return to the agent as the tool response. Search and crawling are also available as the search and crawl CLI commands, crawling also as a Python API.

searxNcrawl exposes three MCP tools:

Tool

What it does

search

Queries the configured providers (SearXNG, You.com) in parallel and returns one merged, deduplicated list with the source of each result. Filters: language, time range, safe search, paging; categories and engines for SearXNG. With include_content, also returns the Markdown of each result page in the same call.

crawl

Renders one or many URLs in Chromium and returns Markdown or JSON with metadata and references.

crawl_site

Follows links from a seed URL within depth, page and timeout limits, tuned for documentation sites.

Both crawl tools drop sections that repeat within a page's Markdown (dedup_mode="exact"), can strip links for a smaller context (remove_links), and accept a Playwright storage_state for pages behind a login. Full parameters are in the MCP tools reference.

Private SearXNG instances

  • SEARXNG_EXTRA_HEADERS adds headers to SearXNG requests in both CLI and MCP, including access-gateway credentials and a custom User-Agent.

  • Values such as @~/.config/searxncrawl/cf_secret load secrets from files; absolute and ~/ paths are supported.

  • Invalid header configuration disables SearXNG with a value-free warning; a configured You.com provider remains available.

  • Setup and JSON examples: SearXNG configuration.

Why searxNcrawl

Hosted search and crawl APIs are quick to start with, but every call goes through someone else's service and is billed per request or credit. searxNcrawl runs on your own machine or server.

searxNcrawl

Firecrawl

Tavily

Exa

Search and crawl run on your hardware

✅ Python package or Docker; You.com queries go to You.com

✅ Self-host stack (Docker Compose), or hosted API

❌ Hosted API

❌ Hosted API

API key

Not needed with SearXNG; You.com needs one

Hosted API: needed. Self-hosted: not by default

Needed

Optional, anonymous use is rate-limited

Cost per call

None with SearXNG; You.com per its plans

Hosted: free monthly credits, then paid credits. Self-hosted: none

Free monthly credits, then paid credits

Free monthly credits, then paid per request

Search backend

Your SearXNG (the engines you enable) and/or You.com

Hosted: Firecrawl's search. Self-hosted: SearXNG if configured, otherwise DuckDuckGo

Tavily's search API

Exa's search index

Licence

MIT

AGPL-3.0

MCP server MIT, API proprietary

MCP server MIT, API proprietary

Self-hosted Firecrawl can search through SearXNG too; it runs as a multi-service stack (API, Playwright service, Redis, RabbitMQ, PostgreSQL and more), while searxNcrawl is a single Python process with Chromium, plus your SearXNG instance if you use one.

Where the others are a better fit: if you do not want to run a search provider and a browser yourself, a hosted API is less work, and Exa's semantic search index or Firecrawl's extraction features go beyond what searxNcrawl does today. Comparison based on each project's public repository and documentation in October 2026; corrections are welcome. Firecrawl is a trademark of its owner; searxNcrawl is not affiliated with it.

FAQ

Only for the search tool, and only if you do not use You.com (YDC_API_KEY). crawl and crawl_site work without any search provider. SearXNG serves HTML only by default, so JSON output must be enabled (search.formats in settings.yml). Public instances may not enable it and often rate-limit automated requests, so run your own. Installation has a Docker one-liner.

Yes. Pages are rendered in Playwright's Chromium through Crawl4AI before they are converted to Markdown, so client-side content is captured.

Yes. Capture a session once with crawl-capture, then pass the saved storage_state file to crawl or crawl_site. See Session capture.

  • Start with --transport http (port 8000 by default) or Docker Compose (port 9555).

  • Put shared access behind an authenticated TLS proxy and connect clients to its /mcp endpoint.

  • Enable header protection and configure the externally visible Host and Origin allowlists.

The first crawl downloads Playwright's Chromium (about 550 MB on disk). Later runs reuse it. Set PLAYWRIGHT_AUTO_INSTALL=false where browsers are provisioned separately.

searxng-mcp is deprecated in favour of searxNcrawl. It only searched SearXNG and could pick a random public instance; searxNcrawl adds page and site crawling but needs your own SEARXNG_URL (or a You.com key), as random public-instance selection is not included.

  • No. Connect directly over STDIO or HTTP for the standard setup.

  • The optional adapter connects a separately hosted worker to Arcade's authentication and gateway.

  • Arcade Cloud's current runner cannot install Playwright; the worker needs a compatible host.

Documentation

Guide

Contents

Installation

uvx, Docker Compose, pip, uv, search providers and environment variables

MCP server

STDIO and HTTP transports, client configuration, Host and CORS guards

MCP tools

Parameters for crawl, crawl_site and search

CLI

crawl, search and crawl-capture

Python API

crawl_page, crawl_site and crawler configuration overrides

Output formats

Markdown and JSON structure, CrawledDocument

Session capture

Authenticated crawling with a saved browser session

MCP Registry

server.json and the publish steps

Arcade adapter (optional)

Shared authentication and routing through Arcade; separate deployment

Architecture

Modules and features for contributors

Contributing

Bug reports, feature ideas and pull requests are welcome. Read the contributing guide and open an issue first for larger changes. Coding agents find project conventions in AGENTS.md. Report vulnerabilities privately as described in the security policy.

License

MIT — © 2026 DDM – Das Digitale Momentum GmbH & Co KG


Available Tools

3 tools
crawlCrawlC

Crawl one or more URLs and extract content as markdown or JSON.

Inputs: urls (required), timeout, dedup_mode, concurrency, +3 more.

crawl_siteCrawl SiteC

Crawl an entire website starting from a seed URL using DFS strategy.

Inputs: url (required), timeout, max_depth, max_pages, +5 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 3 tools

    Disambiguation4/5

    The three tools have mostly distinct roles: search queries providers, crawl fetches specific URLs, and crawl_site traverses a whole site. The crawl vs. crawl_site boundary is clarified by the DFS/seed-URL description, though some overlap remains because crawl accepts multiple URLs.

    Naming Consistency4/5

    All names are lowercase snake_case and action-oriented, which is readable and predictable. However, search and crawl are bare verbs while crawl_site uses a verb_noun compound, so the convention is not perfectly uniform.

    Tool Count5/5

    Three tools are well scoped for a search-and-crawl server: one search entry point, one targeted URL crawler, and one whole-site crawler. Each tool earns its place without obvious redundancy.

    Completeness5/5

    The surface covers the core domain lifecycle: searching configured providers, extracting content from specific URLs, and crawling an entire site from a seed URL. No obvious missing operation prevents agents from accomplishing search-and-crawl workflows.

    Maintenance

    ActivityMaintained
    ResponsivenessMixed handling

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      MCP server for web search and content extraction using DuckDuckGo or SearXNG, with Playwright-based fetching and LLM-powered data extraction.
      138
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      MCP server for internet search via direct Google and DuckDuckGo HTML scraping with AI-powered result normalization and optional summarization, requiring no API keys for search.
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      A privacy-focused web search and content extraction MCP server. It integrates SearxNG with fallback to Google scraping, featuring relevance ranking, security-aware search, and rate limiting.
      3
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      MCP server for local web search via SearXNG, providing unlimited queries without API keys or cost, with automatic fallback to public instances.
      3
      1
      MIT