# keyless-web-search-mcp

Standalone MCP stdio server: the keyless `web_search` tool over a self-operated engine pool (**Bing**, **360**, **Baidu**, **Google**, **Naver**, **Yandex**, **DuckDuckGo**) and — opt-in via `ENABLE_WEB_FETCH=1` — `web_fetch` (anonymous HTTP(S) page reader). Each search keeps at most **two usable engines**, routes default priorities by query language, back-fills failed or irrelevant engines within a bounded attempt budget, relevance-ranks results, merges them, and dedupes by canonical URL. Self-contained directory, zero build step, not part of the harness package system — move it anywhere.

## Why this exists

The harness's built-in `web_search` (DeepSeek provider) needs `DEEPSEEK_API_KEY` and sends every query to the DeepSeek cloud. This server is the local-model-friendly alternative: no keys, no vendor search API, works on mainland-China and HK networks (Bing/360/Baidu from mainland lines, Bing/360/Naver/Yandex from the tested HK line; DuckDuckGo is not reachable from any of them).

`web_fetch` exists for the same reason on the read side: it was built when the harness web profile mounted a search provider but **no fetch provider**, so the built-in `web_fetch` tool failed every call with `WEB_PROVIDER_UNAVAILABLE`. That gap is now filled by the harness, so `web_fetch` registers **only when `ENABLE_WEB_FETCH=1`** is set in the server's env — the default mount is search-only, and a session that already has a harness fetch tool does not see two readers.

**Versioning.** The version in `package.json` / `SERVER_VERSION` is bumped only by the project owner when cutting a release; working-tree changes never bump it.

## Run

```sh
npm install --cache ./.npm-cache   # deps: @modelcontextprotocol/sdk, zod
node index.js                      # speaks MCP over stdio (web_search only)
ENABLE_WEB_FETCH=1 node index.js     # also register the web_fetch reader
npm test                            # run deterministic ranking/fallback tests
```

Quick probe without a client:

```sh
printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"0.0.1"}}}' \
  '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
  | node index.js
```

## Install in Claude Code

Install it for the current user so every Claude Code project can use it. Resolve both paths when registering the server; this avoids depending on the shell's working directory or `PATH` later:

```sh
claude mcp add --transport stdio --scope user web-search-self -- "$(command -v node)" "/absolute/path/to/keyless-web-search-mcp/index.js"
claude mcp get web-search-self
```

Use `--scope local` instead to keep it private to the current project, or `--scope project` to write a shareable `.mcp.json`. Claude Code stores user/local registrations in `~/.claude.json`; MCP server definitions do not belong in `settings.json`.

After registration, `claude mcp list` should report `web-search-self` as connected. The tool appears as `mcp__web-search-self__web_search` (and `mcp__web-search-self__web_fetch` too, when the registration's env sets `ENABLE_WEB_FETCH=1`).

## Tools

### `web_search`

`web_search({ query, count?, maxSources?, engines? })`

| Param | Default | Meaning |
|---|---|---|
| `query` | — | Search query, any language |
| `count` | `8` | Max merged results (1–20) |
| `maxSources` | `2` | Max usable engines per search (hard cap 2; fallback attempts are bounded) |
| `engines` | query-language dependent | Optional explicit pool in priority order; when omitted, Chinese/Korean/Japanese/Russian queries use region-appropriate priorities |

**Two-source policy.** The pool is routed by query language when `engines` is omitted: Chinese starts with 360/Baidu/Bing; English starts with Bing/Google/Naver/360 so the four-attempt budget retains working fallbacks on the tested mainland/HK networks. An explicit `engines` list remains authoritative. Candidates run in bounded parallel rounds; an engine consumes a source slot only after it returns parseable results that pass conservative lexical relevance filtering. Failed, challenged, empty, or irrelevant engines are back-filled up to the attempt and collector budgets, and diagnostics are shown in an "Engine note" line. Results are ranked by query overlap with a small source-quality adjustment: obvious repost/content-farm signals are penalized, while official/docs/education/GitHub signals are favored without hard-blocking ordinary sites. Merging remains **round-robin interleaved** (engine A's 1st, engine B's 1st, A's 2nd, …), and canonical URL dedupe removes fragments, common tracking parameters, and safe `www.` differences. Resilience: an engine that throws a bot-challenge diagnostic enters a 30-minute challenge memory and is skipped — without consuming an attempt — on subsequent searches; if the relevance filter discards every engine's results, the search returns the parsing engines' raw top results with an explicit note instead of failing empty; long queries relax the relevance threshold proportionally (the 0.3 bar holds for up to 4 terms), and boolean words like `or` no longer count as query terms.

Result: numbered list of `title [engine]` / real destination URL / snippet. Link cleanup per engine: Bing click-tracker links decode locally (base64 `u` parameter); 360 reads the `data-mdurl` attribute; Baidu reads the block's `mu` attribute (direct URL, with one best-effort redirect GET only for legacy `link?url=` wrappers); Naver and Yandex titles carry the direct URL in the anchor; Google `/url?q=` wrappers unwrap; DuckDuckGo `/l/?uddg=` trackers unwrap. Per-engine probe budgets: 10 s (Bing, Baidu), 8 s (360, Naver, Yandex), 5 s (Google, DuckDuckGo) — all run in parallel, so the round is only as slow as its slowest member.

### `web_fetch` (opt-in: `ENABLE_WEB_FETCH=1`)

`web_fetch({ url, maxChars? })`

| Param | Default | Meaning |
|---|---|---|
| `url` | — | Absolute `http(s)` URL (anything else is rejected with a diagnostic) |
| `maxChars` | `20000` | Max content characters returned (1000–100000) |

Anonymous public-web read, no credentials: browser User-Agent, at most five redirects, 20 s wall clock covering headers and body, response body cut at exactly 5 MB. Every initial and redirect target is checked after DNS resolution; loopback, private, link-local, carrier-grade NAT, reserved, multicast, and local hostnames are rejected to prevent SSRF into the machine or cloud metadata services. HTML is stripped to visible text (script/style/noscript/svg/head/iframe/canvas/form dropped — an unclosed script/style/noscript/iframe cuts the document there, matching HTML5 raw-text parsing — block boundaries become newlines, the `<title>` is carried in the header), and up to 30 unique in-page links are appended (relative refs resolved against the final URL, fragments and non-http schemes skipped, low-signal search/social hosts dropped) so the model can follow up without re-searching. text/* and XML media pass through entity-decoded using their declared charset; JSON, JavaScript, and form bodies carry no entity layer and pass through byte-exact; binary media types are rejected. A status-header note flags large pages that yield little visible text (likely JavaScript-rendered). Output is a status header — `status`, final `url` (after redirects), `content-type`, truncation — followed by the content. Non-2xx answers return the body head plus a note (403/429 read as bot-check or paywall), never a fabricated page.

## Engines and their status

Verified 2026-07 from three lines (an earlier mainland line, a Shanghai Telecom line, and a Hong Kong Zenlayer datacenter line):

| Engine | Endpoint | Status from these networks |
|---|---|---|
| `bing` | `www.bing.com/search` | ✅ ~10 organic blocks on both lines. Depending on the line, `www` serves the SERP directly or 302s to the `cn.bing.com` twin (the engine follows it); `cn` uses direct result URLs instead of `/ck/a` trackers — both shapes parse |
| `360` | `www.so.com/s` | ✅ clean 200; organic blocks carry the real URL in `data-mdurl` |
| `google` | `www.google.com/search` | Reachable (200) but this client/IP is not trusted for plain-HTML SERPs: the body is a no-JS `enablejs` meta-refresh wall with zero organic results. Detected and skipped with a diagnostic; from a cleaner IP it would join the merge (see the bypass attempts below) |
| `baidu` | `www.baidu.com/s` | ✅ on the Shanghai and HK lines: the risk control gates script-shaped requests (UA-only 302s to the `wappass.baidu.com` image captcha regardless of cookies) but accepts the full document-navigation header signature (`Accept` + `Referer` + `sec-fetch-*` + `Upgrade-Insecure-Requests`) — stable across repeated runs, with or without cookies. The engine sends that signature; lines that gate harder (the earlier mainland line) still hit the detected-and-skipped captcha diagnostic. Parser reads the block's `mu` attribute (direct URL); non-organic cards (recommend lists, hot boards, related search) are skipped like ad slots elsewhere |
| `naver` | `search.naver.com/search.naver` | ✅ on the HK line (200, ~10 `fds-web-doc-root` organic blocks, stable across runs; answers Chinese queries with Chinese results). Title anchor carries the direct URL; the Korean a11y label "새 창 열림" is stripped from titles and snippets. Korean-first index that also covers global technical sites; untested from mainland lines (would degrade to the standard unreachable diagnostic) |
| `yandex` | `yandex.com/search` | ✅ on the HK line: same navigation-signature lesson as Baidu — script-shaped requests get the SmartCaptcha "not a robot" checkbox, but the full document-navigation header set **plus the homepage's cookie session** (yandexuid et al.) is served the plain HTML SERP (~50 `Organic` blocks, direct URLs in `OrganicTitle` anchors). The engine warms the session in-process (30-min TTL, one re-warm retry on a challenged answer). Caveat: from datacenter IPs Yandex runs a rolling rate state — a burst of searches flips the IP back into captcha mode for a few minutes, which is why it sits sixth in the pool (consulted only when the engines ahead of it fail) |
| `duckduckgo` | `html.duckduckgo.com/html` | ❌ TCP-unreachable on all three tested lines (including HK); last-resort engine for networks where it works |

Deliberately not in the pool: **Sogou** — 302s to `sogou.com/antispider/`, the same IP-wall class Baidu covers. Also scanned and rejected from the HK line: Mojeek (serves a captcha page), Ecosia (403 "Ecosia Firewall"), MetaGer (redirects to a non-result page), and Yahoo/Brave/Qwant/Startpage/goo.ne.jp (TCP-unreachable from both mainland and HK lines).

### Google wall: what was tried

The 200 response from this IP is not a block but a **JS challenge interstitial** (~90 KB of obfuscated/encrypted JavaScript; the no-JS path is a meta-refresh into a dead end). The wall turns out to be two layers:

1. **JS challenge (computational)** — decoded, it computes a proof value and sets an `SG_SS` cookie (5-minute expiry), then reloads. This layer is **solvable outside a browser**: running the page's scripts in plain Node.js with a light DOM shim (cookie jar, `navigator`, `Image`, `document`) completed the computation, and the resulting `SG_SS` cookie was accepted once — Google answered 200 and issued the `NID`/`AEC` trust cookies it otherwise never sends to this IP.
2. **Session-pattern layer (behavioral)** — subsequent plain-HTTP follow-ups, including an exact replica of the script's own reload flow (`emsg=SG_REL` + matching `sei`, jar-merged cookies, browser headers), escalate to the `google.com/sorry` 429 anomaly wall. This layer judges the whole session (TLS/HTTP2 fingerprint, request pacing, method mix), which Node's OpenSSL/undici stack does not match.

Client-side levers that were tested and did not change layer 1's answer: the `gbv=1` parameter, cookie warm-up, full browser fingerprint headers, the `retry/enablejs` flow, the `/m` entry, text-browser/feature-phone/IE6/old-Android UAs, a spoofed Googlebot UA (Google validates it), `CONSENT` cookies, POST submission (405), alternate TLDs, and the "click here" `emsg=SG_REL` link without the cookie. Google-proxy alternatives were also unreachable or walled from this network: Startpage and Qwant time out, Mojeek serves a captcha page, Ecosia 403s.

Conclusion: layer 1 is Node-solvable (demonstrated); layer 2 is not, from a non-browser network stack. The only reliable paths to Google from this machine are a clean-IP proxy or a real browser context (headless Chromium); both sit in the same IP/traffic-pattern layer that a scraper cannot honestly out-argue, and repeated probing risks extending the IP's anomaly window — so this server does not attempt them. Google stays a back-fill engine that activates on networks where it will serve a plain-HTML SERP.

## Mount into DeepSeek Harness

Add to your `cordis.yml` (the MCP bridge hot-reloads this entry):

```yaml
- id: mcp-search
  name: '@deepseek-ai/dsh-mcp-client'
  config:
    serverName: search
    transport: stdio
    command: node
    args: ['/absolute/path/to/keyless-web-search-mcp/index.js']
    # env: { ENABLE_WEB_FETCH: '1' }   # opt-in: also register web_fetch
```

The model then sees the tool as `mcp__search__web_search`; with the commented `env` line enabled it also sees `mcp__search__web_fetch`.

## Limitations

- **Search quality is heuristic**: lexical relevance handles obvious off-topic results and Chinese n-grams; internal search-engine pages are excluded; content-farm/repost signals only adjust ordering, not a universal trust judgment. A relevant result is not proof that its claims are correct — fetch and cross-check important facts.
- **Scraping, not an API**: layout changes can break a parser; the failure is loud ("no parseable organic results"), never fabricated. The Google and DuckDuckGo parsers are written from documented SERP structure and are not live-verifiable from this network — tune them the first time their engines actually answer.
- **Rate control**: both public engines tolerate casual use; hammering draws bot challenges.
- **Query leaves the machine** for the consulted engines (that is the price of keyless search); with the two-source cap, at most two of them see a query per search.
- **Fetch is a keyless public-page reader, not a browser or intranet client**: JS-rendered content is invisible to it (same class of limit as the search parsers); pages that 403/429 anonymous clients are reported, not bypassed. Local/private/reserved network targets are deliberately blocked, including redirect destinations.