web-search MCP server

A self-hosted web search MCP server for Claude Code / any MCP client. Multi-engine (CN-direct + international-via-proxy auto-routing), three tools, and research-backed optimizations: RRF rank fusion, SimHash near-duplicate dedup, per-host rate limiting, frequency-aware caching, and TLS/JA3 fingerprint impersonation fallback.

中文文档

Installation

Prerequisites

• Node.js ≥ 18

• Python 3 + curl_cffi (optional, for TLS impersonation on protected sites):

  pip install curl_cffi

• A proxy on 127.0.0.1:7890 (e.g) — set PROXY_URL="" to disable.

Build

git clone git@github.com:nuoyax/web-search-mcp.git
cd web-search-mcp
npm install

Wire into Claude Code

User-level (available in all projects):

claude mcp add web-search -s user -e PROXY_URL=http://127.0.0.1:7890 \
  -- node /absolute/path/to/web-search-mcp/index.js

Or add to .mcp.json (project-level):

{
  "mcpServers": {
    "web-search": {
      "command": "node",
      "args": ["/absolute/path/to/web-search-mcp/index.js"],
      "env": { "PROXY_URL": "http://127.0.0.1:7890" }
    }
  }
}

Verify

node index.js          # start the MCP server (stdio)
node test-smoke.js     # smoke test every engine + fetch

Related MCP server: evo-scry

Highlights

• 6 engines — DuckDuckGo / Bing (international, via proxy) + Bing CN / Baidu / Sogou / 360 (direct). Auto-selected by query language.

• 3 tools — web_search, fetch_url, deep_research (multi-engine fan-out → dedup → rank → fetch → cited report)

• Auto proxy routing — CN hosts go direct, international hosts go through 127.0.0.1:7890; re-decided per redirect hop.

• RRF rank fusion (k=60) across engines — robust without score normalization.

• SimHash dedup — 64-bit, Hamming ≤ 3, merges syndicated copies across hosts.

• Per-host token bucket + exponential backoff — defeats frequency-based anti-bot detection.

• Disk cache with frequency-aware TTL (search 30 min / news 1 h / docs 7 d).

• TLS/JA3 impersonation via curl_cffi for Cloudflare-protected sites (e.g. docs.anthropic.com).

Architecture

flowchart TB
    Client["Claude Code / MCP Client\n(stdio · JSON-RPC)"]

    subgraph Server["web-search MCP Server  (index.js)"]
        T1["web_search"]
        T2["fetch_url"]
        T3["deep_research"]
        Cache["cache.js\n disk KV · frequency-aware TTL"]
    end

    subgraph Core["Core layer"]
        Eng["engines.js\n6 engine adapters"]
        Fetch["fetcher.js\nHTML → markdown\n+ TLS fallback dispatch"]
        Res["research.js\nRRF fusion + SimHash dedup"]
    end

    subgraph HTTP["http.js  (undici primary path)"]
        Proxy["Proxy routing\nCN direct / Intl → 7890\nre-routed per hop"]
        Bucket["Token bucket\n1 concurrent / host\n+120ms interval"]
        Backoff["Exponential backoff\n429 / 5xx retry\n+ Retry-After"]
        Undici["decompress interceptor\n+ manual 3xx loop"]
        Sim["simhash.js\n64-bit Charikar near-dup"]
    end

    TLS["tlsbypass.js → curl_cffi (Python)\nbrowser JA3/JA4 impersonation"]

    CN["CN engines (direct)\nbaidu · bingcn · sogou · so"]
    Intl["International engines (proxy 127.0.0.1:7890)\nduckduckgo · bing"]

    Client <--> Server
    T1 --> Core
    T2 --> Core
    T3 --> Core
    T1 -.cache.-> Cache
    T2 -.cache.-> Cache
    T3 -.cache.-> Cache
    Core --> HTTP
    HTTP -- "403 / 429\nor hard-case host" --> TLS
    HTTP --> CN
    HTTP --> Intl

deep_research pipeline

flowchart TB
    Q(["query"]) --> S1["1. Engine selection\nCJK → CN first · Latin → Intl first\nor engines=[...] / all"]
    S1 --> S2["2. Fan-out search\nall engines in parallel · fault-tolerant"]
    S2 --> S3["3. Baidu redirect resolution\nbaidu.com/link?url= → real URL"]
    S3 --> S4["4. Two-stage dedup\nstage 1: exact URL · stage 2: SimHash Hamming ≤ 3"]
    S4 --> S5["5. RRF ranking\nscore = Σ 1/(60 + rank)"]
    S5 --> S6["6. Fetch top-K\nmax 4 concurrent · token bucket + backoff\nTLS → curl_cffi on 403"]
    S6 --> S7["7. Synthesize report\ncited markdown"]
    S7 --> R(["cited report"])

Principles

1. Auto proxy routing (CN direct / international via proxy)

src/http.js keeps a list of CN-domain suffixes (baidu.com / bing.com / so.com …). On each request the host suffix decides:

• CN domain hit → direct (proxying CN sites is slower and trips their risk control)

• International host → through 127.0.0.1:7890 (works around host-network issues reaching foreign sites)

• On redirect, the route is re-decided per hop: a 301 from an international site to a CN CDN stops using the proxy

• engine.search() may override with forceProxy / forceDirect (e.g. DuckDuckGo must be proxied)

• Set PROXY_URL="" to disable the proxy (e.g. when a global VPN is active)

2. RRF rank fusion (Reciprocal Rank Fusion)

src/research.js. Each result's score = Σ_engine 1/(k + rankᵢ), with k=60.

• Unsupervised: no score normalization needed; robust to engines returning different counts

• Cross-engine consensus: results that appear in multiple engines near the top naturally surface

• Query-term-in-title is only a 1e-4-scale tiebreaker; it never overrides cross-engine consensus

• k=60 is the standard default from Cormack et al. (2009) and ranx.fuse

3. SimHash near-duplicate dedup

src/simhash.js. 64-bit SimHash (Charikar), tokenized with FNV-1a hashing + unigram/bigram.

• Two stages: exact normalized-URL merge first, then SimHash Hamming ≤ 3 as near-duplicate

• The fingerprint excludes the host, so syndicated copies across different hosts merge (the same article republished by different portals, or the same result wrapped differently by Baidu vs Bing)

• Verified: identical text under different hosts → distance 0; reworded → 11; unrelated → 36 — discriminative

4. Per-host token bucket + exponential backoff (anti frequent-access detection)

src/http.js. Per host:

• Concurrency 1 + ≥120 ms between requests (HOST_BUCKETS)

• Transient errors (429/502/503/504) + network failures: exponential backoff 500·2^attempt + jitter, up to 2 retries

• Honors the Retry-After response header

• Different hosts still run in parallel (no throughput loss)

This directly targets the main cause of "frequent-access detection" — high request rate to a single host.

5. Result caching + frequency-aware TTL (incremental)

src/cache.js. Disk JSON KV with TTLs scaled by site update frequency:

• web_search / fetch_url / deep_research return cached results on hit (tagged _cached (age Nmin, ttl Mmin)_)

• Only caches when there are results (empty results aren't cached, so the next call retries)

• CACHE_DISABLED=1 disables; CACHE_DIR overrides the cache directory

• Rationale: frequency-aware incremental crawling (2010) — refresh interval scaled by page change frequency

6. TLS/JA3 fingerprint impersonation fallback (curl_cffi)

src/tlsbypass.js + src/fetcher.js. Node undici's TLS ClientHello differs from a real browser, so heavily-protected sites (Cloudflare-protected docs.anthropic.com etc.) return 403. curl_cffi (Python) can impersonate a real browser's JA3/JA4 fingerprint + HTTP/2 settings.

Strategy:

• Fast path: undici by default (fast, native)

• Known hard-case hosts (docs/platform.anthropic.com etc.): go straight to curl_cffi

• Generic fallback: on undici 403/429 or network error → auto-fallback to curl_cffi; if both fail, report honestly (with impersonated flag)

• Reports tag TLS: impersonated (curl_cffi) or native (undici)

• Requires Python + pip install curl_cffi; without it, degrades to undici-only (main flow unaffected)

• Rationale: TLS Beyond the Browser (ACM IMC 2019) — non-browser TLS clients are fingerprintable; impersonation closes that gap

7. Engine adapters

A query with CJK characters defaults to CN engines first; pure-Latin queries go international first. engine='all' fans out to every engine.

8. Key undici v7 handling

• interceptors.decompress(): auto-decompress gzip/br/deflate (Bing international returns brotli)

• Manual 3xx redirect loop: undici v7's request() no longer accepts maxRedirections, and the redirect interceptor alone isn't enough (per-request opts default to 0 and short-circuit). The manual loop also lets us re-route proxy/direct per hop.

Tools

web_search

{ "query": "anthropic claude api pricing", "num": 8, "engine": "auto" }

• engine: auto (default) | all | duckduckgo | bing | bingcn | baidu | sogou | so

fetch_url

{ "url": "https://example.com/page", "max_chars": 16000 }

Fetch a URL, strip boilerplate, return markdown. Auto-routes proxy/direct by domain; falls back to TLS impersonation on 403.

deep_research

{
  "query": "中国空间站 最新进展",
  "engines": ["bingcn", "baidu"],
  "num_per_engine": 8,
  "fetch_top_k": 4,
  "fetch_chars": 6000
}

Multi-engine fan-out → dedup & RRF rank → fetch top-K → cited markdown report.

Environment variables

Referenced papers

Papers referenced in the implementation (DOIs verifiable; retrieved via the OpenAlex academic API + DuckDuckGo/Bing):

Result fusion / ranking

Near-duplicate detection

Parallel / incremental crawling / rate limiting

Anti-bot / blocking evasion

Full optimization plan and gap analysis: docs/optimization-research.md.

Project structure

web-search-mcp/
├── index.js              # MCP server entry (registers 3 tools, stdio)
├── src/
│   ├── http.js           # HTTP layer + proxy routing + token bucket + backoff
│   ├── engines.js        # 6 engine adapters
│   ├── fetcher.js        # HTML → markdown + TLS fallback dispatch
│   ├── research.js       # RRF fusion + SimHash dedup pipeline
│   ├── simhash.js        # 64-bit Charikar SimHash
│   ├── cache.js          # disk KV with frequency-aware TTL
│   └── tlsbypass.js      # curl_cffi TLS impersonation bridge
├── test-smoke.js         # smoke test
├── docs/
│   ├── optimization-research.md
│   ├── architecture.svg
│   └── pipeline.svg
├── .mcp.json             # project-level MCP registration
└── package.json

Known limitations

• All network requests time out in 20–25 s; nothing hangs.

• Baidu occasionally returns a CAPTCHA interstitial; that engine then errors and deep_research falls back to the other engines.

• Bing international wraps result links in bing.com/ck/a; the real URL is reconstructed from the result's <cite>.

• DuckDuckGo uses the no-key html.duckduckgo.com/html/ endpoint, which requires the proxy.

• TLS impersonation depends on Python + curl_cffi. Without it, falls back to undici-only and hard-case sites (e.g. docs.anthropic.com) return 403; with it, impersonation is automatic.

• The cache expires by TTL without background refresh; the next call after expiry re-fetches. Set CACHE_DISABLED=1 to disable.

License

MIT