Skip to main content
Glama
burakaydinofficial

searxng-deepdive

searxng-deepdive

Tests License: MIT

An MCP server for SearXNG designed for LLM agents doing real research. Four tools with agent-friendly schemas, multi-page result fanout, lightweight URL→Markdown reading, and tool descriptions generated dynamically from the live engine pool of your SearXNG instance.

Why another mcp-searxng?

Existing packages are minimal — most expose a single search(query) tool with no way for the model to ask for more results, target specific engines, or constrain by category. The richer ones bake static descriptions, so the LLM never learns what's actually enabled on this instance. None of them treat agent-tool-selection ergonomics as a design priority.

searxng-deepdive opens those knobs up:

Feature

This

npm mcp-searxng (ihor-sokoliuk)

PyPI mcp-searxng (SecretiveShell)

Engine targeting

✅ via search_on_engines

Category targeting

✅ via search_by_category

Multi-page fanout in one call

✅ via pages: N

❌ (one page per call)

Pagination

✅ via pageno

Compact response trim

✅ via format: "compact"

Dynamic descriptions per instance

✅ live engine list injected

❌ static

❌ static

Validation with cross-tool hints

✅ engine-vs-category, case-insensitive

Zero-result hints

✅ time_range / unresponsive engines / single-engine

URL reader (HTML→Markdown)

✅ with TOC scan + section extraction

✅ basic

Test suite

✅ 102 unit + integration

minimal

Related MCP server: searxng-mcp

Quickstart

Install via npx -y from any MCP client:

{
  "mcpServers": {
    "searxng": {
      "command": "npx",
      "args": ["-y", "searxng-deepdive"],
      "env": { "SEARXNG_URL": "http://127.0.0.1:7979/" }
    }
  }
}

SEARXNG_URL should point at your running SearXNG instance. Need one? The companion repo SearXNG-Compose ships a plug-and-play Docker stack tuned for LLM consumption.

Requirements: Node.js 22 or newer.

Tools

The server registers four tools. The LLM picks among them based on the descriptions below, augmented at startup with the live engine and category list from your instance.

search(query, [...])

Broad web search across the full enabled engine pool. Use when you don't have a specific source preference. Returns merged, deduplicated results across however many engines respond.

search_on_engines(query, engines, [...])

Search using only the specified engines (e.g. ["arxiv", "pubmed", "semantic scholar"]). The tool description registered with the MCP client includes the actual list of engines enabled on your instance — agents don't have to guess names. Validation rejects invalid names with a "did you mean" hint when they look like categories instead of engines.

search_by_category(query, categories, [...])

Search within specific categories — runs every engine tagged with each. Description includes the live category list and which engines belong to each. Same validation: invalid category names produce a clear error that points at search_on_engines when the offending value is actually an engine name.

web_url_read(url, [readHeadings, section, paragraphRange, startChar, maxLength])

Fetch a URL and convert its HTML to clean Markdown. Lightweight HTTP + HTML→Markdown (no headless browser) — handles ~80% of the static-HTML web (Wikipedia, docs sites, blogs, news, GitHub READMEs).

Token-efficient extraction modes (priority order, first set wins):

  • readHeadings: true — return only the heading list as a hierarchical TOC

  • section: "Installation" — return content under matching heading

  • paragraphRange: "3-7" — 1-indexed paragraph slice

  • startChar + maxLength — character window pagination

Recommended workflow for long pages: TOC scan first (readHeadings), then targeted read (section). Far more token-efficient than fetching the full page up front.

If readHeadings comes back with no entries (Reddit threads, comment sections, blog posts that use bold paragraphs instead of <h*> tags), the page is structurally flat — fall through to paragraphRange for sequential sampling, or just fetch without an extraction mode.

web_url_read also accepts JSON, YAML, and TOML content-types directly (spec files, package manifests, registry API responses, CI workflow YAML), so research agents can read these without the HTML-only stub.

For JS-rendered SPAs and bot-protected sites this tool returns minimal/empty content — fall back to a Chromium-backed reader (e.g. Crawl4AI) for those.

Common parameters across all search tools

  • pageno — 1-indexed starting page (default 1)

  • pages — multi-page fanout in one call (1–5, default 1)

  • time_rangeday / week / month / year (warning: not all engines support this; some return empty when set)

  • language — BCP-47 code or all

  • safe_search — 0 / 1 / 2

  • formatcompact (default) or full

Configuration

Env var

Default

Meaning

SEARXNG_URL

http://127.0.0.1:8080

Base URL of the SearXNG instance

Development

git clone <this repo>
cd searxng-deepdive          # or wherever you cloned to
npm install
npm run build                # tsc
npm test                     # vitest
SEARXNG_URL=http://127.0.0.1:7979 npm run probe    # exercise the SearXNG client
SEARXNG_URL=http://127.0.0.1:7979 npm run dev      # start the MCP stdio server

Pointing an MCP client at the source during development

Use tsx to run from src/ directly so you don't need to rebuild on every edit:

{
  "mcpServers": {
    "searxng": {
      "command": "npx",
      "args": ["-y", "tsx", "/absolute/path/to/searxng-deepdive/src/index.ts"],
      "env": { "SEARXNG_URL": "http://127.0.0.1:7979/" }
    }
  }
}

MCP clients cache the subprocess. When you edit code, the running server keeps the old behavior until the subprocess is killed and respawned. Quit the host (LM Studio, Claude Desktop, etc.) fully and reopen — closing the chat window alone usually isn't enough. Symptom of not doing this: a fix you just shipped doesn't appear to take effect.

Testing

npm test

Test coverage spans seven files:

  • normalize-name — case-insensitive name handling

  • validators — engine/category validation with cross-reference hints

  • zero-result-hint — every hint trigger and its inverse

  • trim-to-compact — response trimming + hint inclusion

  • descriptions — anti-pattern regex checks for the description copy that misled real models in earlier versions ("ignored by engines", "Default 'auto'", etc.) — failing build if they reappear

  • searxng-client — HTTP client with MockAgent: malformed JSON, HTML 502 pages, 429 rate-limit handling, multi-page fanout dedup, all-pages-fail throws

  • url-reader — extraction modes + HTTP integration

Design notes

  • Why four tools instead of one with optional engine/category params? Cleaner agent decision-making. With distinct tools the LLM sees explicit purposes; with one fat tool it has to remember when to set which optional flags. Trade-off: more entries in the MCP tool list, mostly identical handler code. Net: better agent ergonomics, especially for smaller models.

  • Why format: "compact" as default? SearXNG's full result objects are several times heavier than just url+title+content+engine. For the typical agent workflow (rank candidates, pick a few to fetch in detail), the compact form is what the LLM actually uses. format: "full" is one parameter away when you need scores, dates, authors, or DOI.

  • Why dynamic descriptions? Static descriptions either list every upstream engine (most aren't enabled on a given instance — wastes context) or list none (LLM has no idea what to put in engines). Live introspection of /config at server startup gives the LLM exactly the right hint for this instance.

  • Why convert silent-wrong into informatively-wrong? Real LM Studio testing showed agents repeatedly stuck in retry loops because failed searches looked successful (zero results, looked like "no matches"; or 60 garbage results, looked like the search ran). The validation + zero-result-hint pattern surfaces the actual cause every time. The description-anti-pattern test suite locks in copy that was empirically shown to mislead models.

Security notes

This package is designed to run locally, inside the user's trust boundary, alongside an MCP-speaking LLM client (Claude Desktop, LM Studio, Cursor, etc.). The trust model assumes:

  • the LLM is acting on the user's behalf

  • the user controls what model is connected to the server

  • the MCP transport is stdio, not exposed to remote callers

Within that boundary, two surfaces are worth knowing about:

  • web_url_read will fetch any HTTP(S) URL the model hands it, with up to five redirects. On a host that can route to private networks, the model can therefore reach intranet services, link-local addresses, or cloud-instance metadata endpoints (169.254.169.254, etc.). This is by design for a local research tool but means you should not run this MCP server in topologies where an untrusted party can pick the URLs (e.g. a hosted MCP gateway facing the public internet). If you do want to lock it down, set SEARXNG_DEEPDIVE_BLOCK_PRIVATE=1 to refuse private / loopback / link-local / metadata destinations — enforced on every redirect hop and against the actual dialed IP, so DNS rebinding can't slip past — and allow specific internal hosts back in with SEARXNG_DEEPDIVE_ALLOWED_HOSTS. Non-HTTP(S) schemes — including file:// — are always rejected, so the tool never reads local files. Body size is capped at 10 MB, a single request has a 45 s total deadline, and the returned Markdown is length-capped (paginate with startChar/maxLength), so a malformed or oversized upstream can't trivially exhaust memory or the caller's context window.

  • The search tools forward the model's query verbatim to SearXNG. SearXNG is the trust boundary for upstream engine traffic; this package does not add additional rate-limiting or query rewriting.

  • Tool output is adversarial input — prompt injection is possible. Search-result snippets and the Markdown returned by web_url_read both contain text the model will read as part of its working context. A page or snippet you don't control can carry instructions ("ignore previous instructions and …"). This isn't a defect in this MCP server — it's inherent to any tool that returns external text — but agent loops that auto-act on tool output without human review are the threat model. Treat tool output as untrusted input, especially for web_url_read against URLs the model picked rather than the user.

This package is provided as-is under MIT with no warranty or liability for damages — see LICENSE. Report suspected vulnerabilities privately via GitHub Security Advisories rather than opening a public issue. See SECURITY.md.

License

MIT — see LICENSE.

Available Tools

4 tools
search_by_categoryA

Search within one or more categories — runs every engine tagged with each (e.g. categories: ["science"]).

Use when you want broad coverage of a content type without enumerating engines.

NOTE: this instance's live engine/category list is currently unavailable — SearXNG was unreachable when the server started. Names you pass are forwarded and validated once SearXNG is reachable; restart the server after SearXNG is up to see the enabled list here.

Multi-page fanout (pages=N) and other knobs work the same as the broad search tool.

ParametersJSON Schema
NameRequiredDescriptionDefault
pagesNoMulti-page fanout: fetch this many consecutive pages in parallel and merge with URL-based dedup. Default 1. Combines with pageno: pageno=3 + pages=2 fetches pages 3 and 4. Diminishing returns past page 2 (engines exhaust their result pools); going wide may also rate-limit upstream engines. Capped at 5 to bound token cost.
queryYesPlain-language search query. Forwarded to each engine's native parser; engine-specific operators (e.g. site:, filetype:) are passed through unchanged.
formatNo'compact' (default) returns only url/title/content/engine for each result — typically much smaller, recommended for ranking and triage. 'full' adds: relevance score, publishedDate, the list of all engines that surfaced the result, and engine-specific metadata (authors, DOI, etc. for academic engines). Switch to 'full' only when you specifically need one of those fields.
pagenoNo1-indexed starting page number. Default 1. Higher pages fetch additional results from each engine, deduplicated against earlier pages by URL. Per-page yield drops sharply after the first 1-2 pages as engines exhaust their result pools.
languageNoLanguage filter, e.g. 'en', 'fr', 'de', 'pt-BR', or 'all' to disable filtering. If omitted, the SearXNG instance's configured default applies (NOT autodetected from the query, despite what the name suggests).
categoriesYesCategory names to constrain the search. SearXNG runs every engine tagged with these categories. The available categories are enumerated in this tool's description above.
time_rangeNoFilter to results published within this time window. WARNING: not all engines implement time-range filtering — some (notably academic engines) return ZERO results when this is set instead of ignoring it. Set time_range only when freshness genuinely matters; if a query returns 0 results with time_range set, retry without it.
safe_searchNoSafe-search level. 0 = off, 1 = moderate, 2 = strict. If omitted, the SearXNG instance's configured default applies (often 0 for self-hosted, but not guaranteed).

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. Discloses that engine/category list is currently unavailable due to SearXNG unreachability, and explains validation behavior. Also describes multi-page fanout and dedup behavior. Sufficient for a search tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is concise at about 10 sentences, front-loading the purpose. The note about server unreachability is important but adds length. Overall no wasted words, well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 8 parameters and no output schema, description covers category behavior, multi-page fanout, error conditions (SearXNG unreachable), and references to sibling tool. Lacks explicit return format but acceptable since no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline 3. The tool description does not add additional meaning beyond what the schema already provides for each parameter; the only extra context is the note about categories availability, which is minor.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly specifies verb 'Search' within 'one or more categories', running engines tagged with each. Distinguishes from siblings by focusing on category-based broad coverage without enumerating engines, and explicitly compares to 'search' tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Describes when to use: for broad coverage of content type without enumerating engines. Also notes caveats about engine list unavailability and how to fix. Implicitly advises against using when engine list is needed, but lacks explicit when-not or comparison to 'search_on_engines'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_on_enginesA

Search using ONLY the specified engines (e.g. engines: ["arxiv", "duckduckgo"]).

Use when you have a specific source preference instead of the broad search tool.

NOTE: this instance's live engine/category list is currently unavailable — SearXNG was unreachable when the server started. Names you pass are forwarded and validated once SearXNG is reachable; restart the server after SearXNG is up to see the enabled list here.

Multi-page fanout (pages=N) and other knobs work the same as the broad search tool.

ParametersJSON Schema
NameRequiredDescriptionDefault
pagesNoMulti-page fanout: fetch this many consecutive pages in parallel and merge with URL-based dedup. Default 1. Combines with pageno: pageno=3 + pages=2 fetches pages 3 and 4. Diminishing returns past page 2 (engines exhaust their result pools); going wide may also rate-limit upstream engines. Capped at 5 to bound token cost.
queryYesPlain-language search query. Forwarded to each engine's native parser; engine-specific operators (e.g. site:, filetype:) are passed through unchanged.
formatNo'compact' (default) returns only url/title/content/engine for each result — typically much smaller, recommended for ranking and triage. 'full' adds: relevance score, publishedDate, the list of all engines that surfaced the result, and engine-specific metadata (authors, DOI, etc. for academic engines). Switch to 'full' only when you specifically need one of those fields.
pagenoNo1-indexed starting page number. Default 1. Higher pages fetch additional results from each engine, deduplicated against earlier pages by URL. Per-page yield drops sharply after the first 1-2 pages as engines exhaust their result pools.
enginesYesEngine names to use, lowercase. Multiple engines run in parallel and results are merged. Must match this instance's enabled engines (the available list is enumerated in this tool's description above).
languageNoLanguage filter, e.g. 'en', 'fr', 'de', 'pt-BR', or 'all' to disable filtering. If omitted, the SearXNG instance's configured default applies (NOT autodetected from the query, despite what the name suggests).
time_rangeNoFilter to results published within this time window. WARNING: not all engines implement time-range filtering — some (notably academic engines) return ZERO results when this is set instead of ignoring it. Set time_range only when freshness genuinely matters; if a query returns 0 results with time_range set, retry without it.
safe_searchNoSafe-search level. 0 = off, 1 = moderate, 2 = strict. If omitted, the SearXNG instance's configured default applies (often 0 for self-hosted, but not guaranteed).

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description must convey behavioral traits. It discloses the current state (SearXNG unreachable) and that engine names are forwarded and validated later. However, it does not describe behavior for invalid engines, error handling, or rate limits, leaving gaps in transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is relatively concise with two paragraphs. The first sentence is direct. The note about the engine list is necessary context. Slightly verbose with the restart instruction, but overall efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite 8 parameters and no output schema, the description covers the primary use case, current instance state, and comparison with sibling tools. Missing explanation of result format, but acceptable for a search tool with sibling context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds a usage example but no new meaning beyond what the schema provides. No reduction or increase warranted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: searching using specified engines only, with an example. It explicitly differentiates from the broad `search` tool, making the purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit guidance: 'Use when you have a specific source preference instead of the broad `search` tool.' Also mentions the alternative tool and notes the current unavailability of the engine list, with restart instructions. This provides clear when-to-use and when-not-to-use context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_url_readA

Fetch a URL and convert its HTML content to clean Markdown.

Use after search (or its variants) when you have a URL and want the actual page text, not just the search snippet. Lightweight HTTP + HTML→Markdown — handles ~80% of the static-HTML web (Wikipedia, docs sites, blogs, news, GitHub READMEs).

What this DOES NOT handle: • JavaScript-rendered pages (React/Vue/Angular SPAs) — content loads after the initial HTML, which we don't execute. Returns minimal or empty markdown for these. • Bot-protected pages (Cloudflare challenge, captcha) — typically fail with HTTP 403/503. • Binary resources (PDF, images, archives) — returns an explanatory hint instead of garbled bytes.

For those cases, fall back to a Chromium-backed reader (e.g. Crawl4AI exposed via the SearXNG-Compose reader profile).

Token-efficient extraction modes (priority order — first one set wins): • readHeadings:true — returns ONLY the heading list (hierarchical TOC). Cheapest survey of a long page. • section:'' — returns content under first matching heading, up to the next same-or-higher heading. Use after readHeadings to jump. • paragraphRange:'3-7' — 1-indexed paragraph slice; supports 'N' (single) or 'N-M' (range). • startChar + maxLength — character window pagination. Response includes total_length and truncated so you can plan follow-up calls.

If no extraction mode is set, returns the full Markdown.

Recommended workflow for long pages:

  1. web_url_read(url, readHeadings: true) ← TOC scan

  2. web_url_read(url, section: '') ← targeted read Far more token-efficient than fetching the full page up front.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesHTTP(S) URL to fetch and convert to Markdown. Static pages only — for JavaScript-rendered SPAs and bot-protected sites, use a Chromium-backed reader instead.
sectionNoSubstring match against headings (case-insensitive). Returns content under the FIRST heading containing this text, up to the next heading at the same or higher level. Use after readHeadings to read a targeted section without dumping the whole page.
maxLengthNoMaximum characters to return. Use with startChar for paginated reading; the response includes `total_length` and `truncated` so you can plan follow-up calls.
startCharNoCharacter offset where extraction begins. Default 0.
readHeadingsNoWhen true, returns ONLY the page's heading list as a hierarchical TOC (token-cheap survey). Pair with a follow-up call using `section` to read the chunk you actually want.
paragraphRangeNo1-indexed paragraph range, syntax 'N' or 'N-M' (e.g. '3-7' for paragraphs 3–7, '5' for paragraph 5 alone). Useful for sequential reading of long pages without re-fetching.

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, description fully covers behavior: limitations (JS-rendered, bot-protected, binary), response fields (total_length, truncated), and extraction mode priority. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with bullet points and sections. Slightly long but every sentence adds value. Front-loaded with purpose and use cases.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite no output schema, description adequately explains response fields (total_length, truncated). Covers all 6 parameters, usage patterns, and limitations. Complete for a complex tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, providing baseline of 3. The description adds value by explaining priority order, use of section matching, and workflow integration, going beyond schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it fetches a URL and converts HTML to Markdown. It distinguishes from sibling tools like search by specifying it provides actual page text, not snippets. Explicitly lists what it does and does not handle.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit context for use: 'Use after search' and suggests fallback to Chromium-based reader for problematic pages. Details priority order for extraction modes and a recommended workflow for long pages.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 4 tool updatesv0.4.1
    • First observedsearch
    • First observedsearch_by_category
    • First observedsearch_on_engines
    • First observedweb_url_read

TDQS

A4.4/5.0

Scored across 4 tools

Disambiguation5/5

Each tool has a clearly distinct purpose: 'search' uses all engines, 'search_on_engines' specifies engines, 'search_by_category' uses categories, and 'web_url_read' fetches URL content. No overlap or ambiguity.

Naming Consistency5/5

All tool names follow a consistent 'action_noun' pattern in snake_case. The three search tools share the 'search' base with clear qualifiers, and 'web_url_read' follows the same convention. Perfectly predictable.

Tool Count5/5

4 tools is ideal for a search-focused MCP server: three search variants covering broad, specific, and category-based searches, plus a URL reader. No unnecessary tools, and no obvious missing core functionality.

Completeness4/5

The tool set covers the main search and fetch operations well. Minor gap: there's no tool to list available engines or categories, but this is noted as a server startup issue rather than a missing tool. The core lifecycle is complete.

Maintenance

ActivityStale
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • F
    license
    A
    quality
    A
    maintenance
    An MCP server for SearXNG that provides web search capabilities with concise model-visible output while preserving full result payloads in metadata. It supports search, parallel fetching, URL extraction, and research workflows through both local stdio and streamable HTTP transports.
    7
    2
    -
  • A
    license
    A
    quality
    B
    maintenance
    An MCP server that integrates the SearXNG API for web search and URL content extraction with advanced features like pagination, caching, and proxy support.
    4
    6,608
    2
    MIT