searxng-mcp-scraper

MCP server exposing SearXNG search and HTTP fetch tools for AI agents.

A self-contained Model Context Protocol server that turns SearXNG (a self-hosted metasearch engine) into two MCP tools you can wire into Claude Desktop, Cursor, or any MCP-compatible client:

• search(query, ...) — ranked web results from SearXNG's JSON API

• fetch(url) — cleaned main text of any public http(s) URL

No API keys, no third-party tracking, no rate limits beyond what your SearXNG instance imposes.

Installation

# From source (recommended during dev)
git clone https://github.com/your-org/searxng-mcp-scraper
cd searxng-mcp-scraper
uv sync --extra dev
uv pip install -e .

# Or with plain pip
pip install -e .

Related MCP server: searxng-mcp

Configuration

The server reads configuration from environment variables (or a .env file at startup):

Note: your SearXNG instance must have json enabled in search.formats under settings.yml, or search will return searxng_unavailable with a 403.

PDF extraction (optional, local, keyless)

deep_scrape downloads linked PDFs and extracts their text. By default it uses markitdown. For a much faster and cleaner text extraction on text-based PDFs, install the optional extra:

uv sync --extra pdf-extras          # or: pip install "searxng-mcp-scraper[pdf-extras]"

This pulls in pdf-inspector — Firecrawl's open-source Rust PDF engine (no API key, no cloud, runs locally). When installed, PDF extraction routes through it first:

• text-based PDFs are read straight from the PDF internals (fonts, text operators) in milliseconds.

• scanned / image-only PDFs are correctly classified as needing OCR (the library reports pdf_type: scanned). It has no OCR of its own, so those fall through to markitdown (which may have an OCR backend); if markitdown also returns nothing, the document surfaces an honest DocumentExtractionFailed instead of a silent empty string.

The dependency is optional: without it, PDFs simply use markitdown. Nothing in the core package depends on a native wheel.

Firecrawl fallback (JS-rendered pages)

fetch does a plain HTTP GET and runs trafilatura over the HTML. That is fast and free, but it returns nothing useful for client-rendered pages: the served body is an empty root div, so trafilatura scrapes up the meta description and a nav label and returns a couple hundred characters of noise. Measured on real sites:

When a fetch errors, returns empty text, or returns fewer than FIRECRAWL_MIN_TEXT_CHARS, the URL is retried once through Firecrawl's /v2/scrape, which renders the page server-side and returns clean markdown. Results carry extractor: "firecrawl" so you can tell which path produced them.

Firecrawl's hosted API has a keyless free tier for /v2/scrape and /v2/search, so this costs nothing and needs no account. The catch: the tier rejects any Authorization header, while most clients insist on sending one. FIRECRAWL_BASE_URL therefore points at a small local proxy that strips the header:

python3 scripts/firecrawl_keyless_proxy.py   # 127.0.0.1:8788

The proxy is vendored here (no external dependency) and forwards a real key if you set FIRECRAWL_UPSTREAM_KEY in its environment — which unlocks /v2/map, /v2/crawl, and /v2/extract through the same URL. Key-gated paths without a key get a clear error rather than a confusing upstream 403.

Set FIRECRAWL_API_KEY and point FIRECRAWL_BASE_URL at https://api.firecrawl.dev to skip the proxy and use a real account instead.

The fallback never overrides a guard. invalid_url and private_network_blocked are SSRF/scheme decisions, not extraction failures — routing those through a third-party renderer would defeat the guard, so they are excluded. oversize is excluded too (re-rendering the same huge page just burns the cap again). If the fallback fails or returns less text than the static path, the original result is kept unchanged. Set FIRECRAWL_FALLBACK_ENABLED=false for fully air-gapped runs.

Running

The server supports two transports. Pick the one your MCP client speaks:

stdio (default — Claude Desktop, Cursor, etc.)

# From a checkout, with .env in the cwd
SEARXNG_BASE_URL=http://localhost:8888 searxng-mcp-scraper

# Or via uv
SEARXNG_BASE_URL=http://localhost:8888 uv run searxng-mcp-scraper

# Or as a module
SEARXNG_BASE_URL=http://localhost:8888 python -m searxng_mcp_scraper

The server speaks MCP over stdio. It writes logs to stderr; stdout is reserved for the JSON-RPC stream and must not be polluted.

streamable-http (llama-ui, Open WebUI, browser-based clients)

SEARXNG_BASE_URL=http://localhost:8888 uv run searxng-mcp-scraper --transport streamable-http

This binds to http://127.0.0.1:8765/mcp by default. Override with MCP_HOST / MCP_PORT env vars. CORS is pre-configured for http://localhost:* and http://127.0.0.1:* — any browser on your machine can connect.

If you need to bind to 0.0.0.0 (LAN), set MCP_CORS_ORIGINS to a narrower list. There is no auth in v1 — do not expose this to the public internet without putting it behind a reverse proxy with auth.

llama-ui / Open WebUI config

For browser-based MCP clients, run the server in HTTP mode and paste the URL into the client's "MCP server URL" field:

1. In one terminal:

   SEARXNG_BASE_URL=http://localhost:8888 uv run searxng-mcp-scraper --transport streamable-http

2. In the UI, add a new MCP server with URL http://127.0.0.1:8765/mcp (no auth headers required for local).

3. The search and fetch tools will appear in the model's tool list.

Claude Desktop config

Add this to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "searxng-scraper": {
      "command": "searxng-mcp-scraper",
      "env": {
        "SEARXNG_BASE_URL": "http://localhost:8888"
      }
    }
  }
}

If you installed from source and the binary isn't on your PATH, point command at uv:

{
  "mcpServers": {
    "searxng-scraper": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/searxng-mcp-scraper", "run", "searxng-mcp-scraper"],
      "env": {
        "SEARXNG_BASE_URL": "http://localhost:8888"
      }
    }
  }
}

Restart Claude Desktop; you should see search and fetch tools appear.

Tool reference

search(query, ...) -> {results, suggestions, number_of_results}

fetch(url) -> {url, final_url, content_type, text, byte_count}

Strips <script>, <style>, <nav>, <header>, <footer>, <aside> blocks via trafilatura, and collapses whitespace. Refuses non-http(s) schemes and (by default) private network targets. Returns a structured error on timeout, oversize, or non-2xx — never crashes.

scrape_blog(blog_url) -> {output_path, post_count, ...}

Discovers a blog's RSS/Atom feed first, falls back to scanning HTML index pages, fetches post text, and writes one Markdown file under BLOG_SCRAPE_OUTPUT_DIR.

deep_scrape(blog_url) -> {output_path, assets_dir, documents_*, images_*}

Does everything scrape_blog does, plus:

• discovers linked documents from each post (.pdf, .docx, .csv, .md, etc.)

• downloads raw document files into deep_assets/

• extracts document text with MarkItDown and embeds it in the Markdown output

• discovers images from <img src>, lazy-load attributes, srcset, and direct image links

• downloads bounded image files into deep_assets/images/

• records per-image URL, saved path, content type, byte count, and any failure in the Markdown

The returned summary includes images_found, images_saved, and image_byte_count. One bad image/document is recorded in-band and does not fail the whole blog scrape.

Development

uv run pytest          # run the full test suite
uv run mypy src        # static type checks
uv run ruff check      # lint

License

MIT.