Skip to main content
Glama

Web Search MCP

Multi-source web search MCP server with RRF fusion, 4-layer URL extraction, and provider health tracking.

Status

Phase 2: 5 providers (SearXNG/ddgr/Tavily/Exa/opencli-zh), RRF fusion, snippet cleaning (rules + optional LLM compression), search/extract/doctor tools, provider health tracking with auto-degradation.

Related MCP server: web-mcp

Install

Add to Claude Desktop 3P config (~/Library/Application Support/Claude-3p/configLibrary/<uuid>.json):

{
  "name": "web-search",
  "source": "user",
  "transport": "stdio",
  "command": "node",
  "args": ["/Users/hades/projects/web-search-mcp/bin/cli.js"],
  "env": {
    "SEARXNG_URL": "http://localhost:18443",
    "TAVILY_API_KEY": "...",
    "EXA_API_KEY": "...",
    "FIRECRAWL_API_KEY": "...",
    "WSM_LLM_PROVIDER": "haiku",
    "ANTHROPIC_API_KEY": "..."
  },
  "toolPolicy": {
    "search": "allow",
    "extract": "allow",
    "doctor": "allow"
  }
}

Once published to npm, replace command: "node" + absolute path with command: "npx" + args: ["-y", "@thehappyboy/web-search-mcp@latest"].

Configuration

Environment variables (or .env in cwd or ~/.agents/skills/web-search/.env):

Var

Default

Required

Description

SEARXNG_URL

http://localhost:18443

no

SearXNG base URL

TAVILY_API_KEY

—

no

Enables Tavily provider

EXA_API_KEY

—

no

Enables Exa provider

FIRECRAWL_API_KEY

—

no

Enables Firecrawl in extract pipeline (layer 3)

WSM_LLM_PROVIDER

none

no

haiku for Anthropic, local for Ollama, none to disable

ANTHROPIC_API_KEY

—

no

Required when WSM_LLM_PROVIDER=haiku

WSM_LOCAL_LLM_URL

—

no

Required when WSM_LLM_PROVIDER=local

WSM_MAX_RESULTS

10

no

Default max results

WSM_TIMEOUT_MS

25000

no

Global search timeout

Providers without keys are automatically disabled. The search tool still works with whatever is available. ddgr, opencli, defuddle are auto-detected via which.

Tools

{
  query: string,              // required
  max_results?: number,       // 1-30, default 10
  category?: 'general'|'news'|'science',
  mode?: 'fast'|'quality',    // quality = LLM compression (top-5 results)
  providers?: string[],       // override default list (replace semantics)
}

Returns:

{
  count: number,
  took_ms: number,
  providers_used: string[],     // providers that returned ≥1 result
  providers_failed: string[],   // providers that errored OR returned []
  results: [{
    title, url, snippet,
    providers: string[],        // which sources surfaced this URL (multi-source corroboration)
    rrf_score: number,          // reciprocal rank fusion score
  }]
}

opencli-zh auto-routing: When query contains keywords like 知乎, 小红书, 雪球, B站, 微博, or patterns like 600519.SH, the opencli-zh provider routes to the corresponding adapter. Force a specific site with opencli_sites (passed via providers extension, see orchestrator).

extract

{
  url: string,                 // required
  timeout?: number,            // 5-60 seconds, default 20
}

Tries 4 layers in order until one succeeds:

  1. defuddle — fastest, pure-text sites

  2. opencli adapter — for known sites (zhihu/xiaohongshu/xueqiu/bilibili/weibo)

  3. Firecrawl — JS-rendered pages (requires FIRECRAWL_API_KEY)

  4. opencli browser — last resort, drives real Chrome

Returns:

{
  content: string,             // markdown
  source: 'defuddle'|'opencli'|'firecrawl'|'opencli-browser'|'none',
  took_ms: number,
  error?: string,              // only when source === 'none'
}

doctor

{} // no arguments

Returns provider availability + runtime health stats:

{
  providers: {
    searxng:    { available: true, url: "http://localhost:18443" },
    ddgr:       { available: true },
    tavily:     { available: true },
    exa:        { available: true },
    opencli_zh: { available: true },
    firecrawl:  { available: true },
    llm:        { available: true, provider: "haiku" },
  },
  health: {
    tavily: {
      total: 24,
      success: 15,
      fail: 9,
      success_rate: 0.625,
      last_failure: "2026-07-06T...",
      last_failure_reason: "timeout",
      recommendation: "healthy" | "insufficient" | "degraded",
    },
    // ...
  },
  version: "0.2.0",
}

When a provider's success_rate drops below 0.5 with at least 20 samples, it's automatically skipped (recommendation: degraded). Health stats are in-memory and reset on restart.

Development

npm test                  # all tests (unit + integration, 158 total)
npm run test:unit         # unit only
npm run test:integration  # integration only (needs searxng at SEARXNG_URL)
npm start                 # start stdio server

Architecture

MCP client (Claude/Cowork)
  │ JSON-RPC over stdio
  ▼
server.js ───┬── search tool ──→ search.js (orchestrator)
             ├── extract tool ─→ extract/index.js (4-layer pipeline)
             └── doctor tool ──→ health tracker snapshot

search.js:
  Provider layer (parallel, isolated failure)
    searxng | ddgr | tavily | exa | opencli-zh
  ▼
  Merger layer: normalize → dedup(URL) → RRF(k=60)
  ▼
  Cleaner layer: rules (default) | LLM compression (quality mode, top-5)
  ▼
  Health tracker records success/failure per provider

extract/index.js:
  defuddle → opencli adapter → Firecrawl → opencli browser
  (each layer returns null on failure, falls through to next)

Phase history

  • Phase 1 (15 tasks): 4 providers (searxng/ddgr/tavily/exa), RRF fusion, rule-based cleaning, stdio MCP, search tool only

  • Phase 2 (8 tasks): opencli-zh provider, LLM quality mode, 4-layer extract, doctor + health auto-degradation

License

MIT

Available Tools

6 tools
doctorC

Diagnose provider availability and health.

Inputs: no named parameters.

extractA

Extract clean markdown from a URL. Tries defuddle → opencli adapter → Firecrawl → opencli browser in order.

Inputs: url (required), timeout.

vertical_batchC

Up to 5 vertical queries in parallel via AnySearch.

Inputs: queries (required).

vertical_sub_domainsA

Discover sub-domains and required parameters for a given vertical domain. Call before vertical_search when sub_domain schema is unknown.

Inputs: domain, domains.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 6 tools

    Disambiguation4/5

    Most tools are clearly distinct: 'doctor' for health checks, 'extract' for URL content, 'search' for general search, and 'vertical_*' for domain-specific search. However, 'vertical_search' and 'vertical_batch' both perform vertical searches, differing only in batch size, which could cause slight confusion.

    Naming Consistency3/5

    Naming is inconsistent: 'doctor', 'extract', 'search' are single verbs without a clear pattern, while 'vertical_batch', 'vertical_search', and 'vertical_sub_domains' use a snake_case prefix. This mix of conventions reduces predictability.

    Tool Count5/5

    With 6 tools, the set is well-scoped for a web search and extraction server. Each tool serves a distinct purpose without being excessive or insufficient.

    Completeness4/5

    The tool surface covers core functionality: health checks, general search, URL extraction, and vertical domain search with discovery. Minor gaps exist, such as no tool for image or news search, but the provided set is functionally complete for the stated domain.

    Maintenance

    ActivityInactive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Multi-engine aggregated search MCP server that combines results from 7 search engines with deduplication, relevance ranking, and web page content extraction.
      1
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.
      5
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      MCP server that provides web search via LLM API gateways with server-side search, returning cited answers and multi-provider fallback.
      -