🔍 MyWebSearch

Keyless Multi-Engine AI Web Search & High-Purity Content Extraction Engine
MCP Server · CLI · Local HTTP Daemon · Skill-Guided Agent Workflows

🇨🇳 简体中文 | 🇺🇸 English

One server, four ways to use it: connect an AI client over MCP, run a one-shot CLI query, keep a local daemon alive, or use the built-in diagnostics before going live.

Jump to: Quick start · MCP tools · CLI · Configuration · Diagnostics · Development

✨ Features

my-websearch is a full-stack search and technical documentation engine built specifically for AI coding agents (Claude Desktop, Cursor, Cherry Studio, Windsurf, Cline, ZCode, etc.). Out of the box, ready for multiple mainstream AI clients:

• 🌐 16-Engine Federated Search & Smart Routing
  Aggregates direct domestic engines (Bing, Baidu, CSDN, Juejin, Sogou), global engines (DuckDuckGo, Brave, Startpage, Yahoo, Exa, Bing News), and specialized developer platforms (GitHub, StackOverflow, Hacker News, arXiv). Features automatic language-aware routing (auto), parallel multi-query execution (queries: string[]), URL deduplication, and automatic minResults cascading fallback. When DuckDuckGo is unhealthy, Yahoo automatically joins the English primary group from the fallback pool.

• ⚡ Native Web Protocol & Proof-of-Work Solvers (Browser-Free Fast Path, Browser Fallback Only When Needed)
  Built on wreq-js with native modern TLS/HTTP2 handshake emulation and persistent session cookies. Includes pure-JS/Rust solvers for Startpage's Anubis SHA-256 PoW challenge and DuckDuckGo's d.js (isJsaChallenge) HTML5/arithmetic challenge, so most challenges resolve without launching a browser. Request stacks are engine-specific: DuckDuckGo requires the wreq Chrome profile via GET (axios POST is challenged with HTTP 202), while Brave defaults to curl HTTP/1.1 (Windows Schannel; tested curl 7/10 before rate limiting, axios 0/10, wreq HTTP/2 429). When a Brave 429 turns out to be an anti-bot challenge page, the engine retries once through a disguised headless CDP browser that clicks the Argon2 PoW button itself (~4–6s; measured 8/8 while plain HTTP stayed 429). Note: DuckDuckGo's page-level HTTP 202 anomaly CAPTCHA ("select all squares containing a duck") is human verification and is intentionally not bypassed — it is reported as bot_challenge; Brave's challenge is computational (Argon2 PoW) and is solved by the page's own JavaScript, not by us. Real URL resolution for Sogou redirect links, Baidu Location, and Bing Base64 links.

• 📄 High-Purity Markdown Article Extraction & Dedicated Site Purifiers
  Combines Mozilla Readability with intelligent chrome noise stripping. Features dedicated purification rules for WeChat Official Accounts (mp.weixin.qq.com), StackOverflow questions & accepted answers, and transparent internal routing for CSDN & Juejin. Removes <nav>, <aside>, <footer>, and breadcrumbs while faithfully preserving article titles, code blocks, and GFM tables. Supports seamless pagination (startIndex), plus three token-saving modes: BM25 relevance-ranked focus, CSS-selector extract to JSON with no LLM, and numbered citations.

• 📚 Unified Context7 Library Docs Lookup (queryDocs)
  Directly query official documentation with natural package names (e.g. "next.js", "vue") or Context7 IDs without needing a separate resolve step. Includes automatic candidate ranking, blind-guess 404 self-healing, and related alternatives discovery.

• 🐙 Extended GitHub / Gitee Resource Fetcher
  fetchGithubContent (formerly fetchGithubReadme) supports both GitHub and Gitee: repository READMEs, any source file (/blob/..., with lines/#L120-L180 slicing), directory trees (/tree/...), complete issue / PR discussions, commits (/commit/<sha>), compare views (/compare/a...b), and releases (/releases, /releases/tag/<tag>).

• 🌏 Zero-Config Network Detection & Fake-IP Ready
  Auto-detects OS system proxies (Windows Registry / macOS scutil / Linux env). Automatically routes overseas engines through proxy while keeping domestic engines on fast direct lines. Pre-whitelists 198.18.0.0/15 for TUN / Fake-IP compatibility while enforcing strict SSRF security bounds. No proxy is the supported default: outside mainland China, leave PROXY_URL unset and every engine is queried directly — you don't need Clash/v2ray to use this server.

• 🔌 Versatile Deployment Modes
  Works seamlessly across MCP STDIO (Claude Desktop/Cursor), Streamable HTTP / SSE (Cherry Studio/remote clients), CLI one-shot commands, and a Local HTTP Daemon with connection pool reuse and Prometheus /metrics monitoring.

Related MCP server: web-mcp

🌍 16 Search Engines

Version note: this README tracks the current source tree (1.5.1). 1.5.1 is a search-quality release: diversity-aware early return (usable results + ≥2 engines or ≥3 domains), host cap (≤2 per domain, ≤3 for official docs), Chinese-tech routing to Bing/Baidu/Sogou, entity/title relevance guards, official-intent anchor injection (MCP/Vitest/Node.js/Next.js/React/Vue/TypeScript/Docker/PostgreSQL/Zustand) plus an official LTS page for Node.js version queries, Baidu real-URL extraction, version-aware ranking for date-based specs, a 3.5 s cascade cap / 3.5 s CSDN timeout, apiKeys.gitee support, fast-fail codeSymbol without GITHUB_TOKEN, and an npm run eval:quality 10-scenario benchmark. 1.5.0 unified the test suite (all legacy scripts migrated to Vitest, single runner in CI), centralized every process.env read behind src/config/env.ts (enforced by ESLint), splits the four largest modules, adds a 180 s browser-only window after a Brave 429, lazy-loads the server stack for a ~0.32 s CLI cold start, adds doctor --network cache-hit-rate / circuit-remaining / P50-P95 metrics, upgrades core dependencies (MCP SDK 1.32, axios 1.21, cheerio 1.2, express 5.3, zod 4), and requires Node.js ≥22 (Vitest 5 and the optional unpdf PDF parser both require it; node:sqlite is used when available, with a sharded-JSON fallback) plus a Windows CI job on Node 22. Earlier: 1.4.2 added Brave 429 challenge detection (captcha vs quota) and the disguised CDP browser fallback; 1.4.1 fixed Windows curl 8.4 compatibility; 1.4.0 added codeSymbol, CDP device profiles and DDG 202 fallback. npm latest may lag until the package is published.

🛠️ 5 Core MCP Tools (Minimal & Token-Efficient)

💡 Token Budget & Slimming: All tool descriptions are strictly kept within 250 characters (total description characters ≤ 1200), while detailed routing guidance is delegated to McpServer({ instructions }), significantly reducing LLM context consumption.

💡 Backward Compatibility: Legacy fetchCsdnArticle and fetchJuejinArticle are automatically routed under fetchWebContent; legacy resolveLibraryId is merged into queryDocs (which accepts a library name or the legacy libraryId alias). The separate legacy tools were removed — the REGISTER_LEGACY_ARTICLE_TOOLS switch is gone, because keeping three unregistered handlers around still cost tool-description budget for nothing.

🧩 MCP Prompts: deep-research (multi-source research with citations) and code-lookup (official docs + source + release notes) are registered as server-side prompt templates; clients that support MCP Prompts can invoke them directly.

🗂️ MCP Resources: read-only websearch://cache/pages / websearch://cache/docs indexes plus per-entry Markdown resources expose already-cached content without another tool call. Resources never trigger network fetches.

🚀 Quick Start

1. Run with NPX (Zero Configuration)

No installation required:

npx -y my-websearch@latest

💡 Automated Proxy Discovery: The server automatically detects your operating system's proxy settings. If a local proxy environment is active, overseas engines automatically use the proxy while domestic engines stay on fast direct lines. 198.18.0.0/15 is whitelisted out of the box for Fake-IP environments.

2. Configure in AI Clients

🔹 Automated setup (recommended)

init detects installed MCP clients, writes the server entry into their config files, and keeps a .bak backup:

npx -y my-websearch@latest init                      # auto-detect and write
npx -y my-websearch@latest init --dry-run            # preview the diff, write nothing
npx -y my-websearch@latest init --client cursor,code # target specific clients

Supported out of the box: Claude Desktop, Cursor, and VS Code (workspace .vscode/mcp.json). An existing entry is never overwritten unless you pass --force, corrupt config files are left untouched, and on Windows it writes cmd /c npx for you. Restart the client, then verify with my-websearch doctor.

🔹 Manual configuration

Claude Desktop / Cursor / Windsurf / Cline (STDIO Mode)

Add to your MCP settings file:

{
  "mcpServers": {
    "my-websearch": {
      "command": "npx",
      "args": ["-y", "my-websearch@latest"],
      "env": {
        "MODE": "stdio"
      }
    }
  }
}

(For Windows: init handles this automatically. When configuring manually, if npx is not in the system path, set command to cmd and args to ["/c", "npx", "-y", "my-websearch@latest"])

🔹 Cherry Studio (Streamable HTTP Mode)

Start the server in your terminal with npx my-websearch@latest (default port 3211), then configure:

{
  "mcpServers": {
    "web-search": {
      "name": "MyWebSearch",
      "type": "streamableHttp",
      "baseUrl": "http://localhost:3211/mcp"
    }
  }
}

💻 CLI & Local Daemon

my-websearch can be used directly from your terminal as a standalone CLI tool or persistent daemon. Use --json in automation and keep the daemon on 127.0.0.1 unless you explicitly need remote access:

# Global install
npm install -g my-websearch

# Detect installed MCP clients and write their config (--dry-run to preview)
my-websearch init

# Start persistent local HTTP daemon (port 3210 by default)
my-websearch serve

# One-shot terminal search (reuses running daemon automatically)
my-websearch search "Model Context Protocol best practices" --limit 5 --min-results 5 --json

# Only results published within the last week (day|week|month|year)
my-websearch search "typescript 5.9 release" --time-range week --json

# Extract web page as clean Markdown
my-websearch fetch-web "https://blog.vuejs.org/posts/vue-3-5" --max-chars 15000 --json

# View daemon status and Prometheus metrics (/health, /status, /metrics)
my-websearch status --json

# Clear in-memory search cache
my-websearch cache-clear

🩺 Diagnostics

Use the built-in doctor command before troubleshooting a proxy, browser, or engine:

# Local configuration and browser checks (path only; use --browser to verify launch)
my-websearch doctor

# Verify the Chromium-based browser can actually launch (auto-detected, CHROME_PATH, or PLAYWRIGHT_CDP_ENDPOINT)
my-websearch doctor --browser

# Real proxy/engine probes (may contact upstream services)
my-websearch doctor --network

# Limit network probes to selected engines
my-websearch doctor --network --engine bing,duckduckgo --json

# Reset and clear cached engine circuit breaker / health state
my-websearch doctor --reset-health

The report separates configuration, proxy TCP reachability, HTTP reachability, application parsing, browser launch, and challenge verification status. For Brave, a 429 is additionally classified as captcha (human verification — retrying usually stays challenged) or quota (rate-limit window) in both the text report (challenge=...) and --json (challengeKind). Network mode also summarizes cache hit rates (search cache and fetch page cache), remaining circuit cooldowns per engine, and P50/P95 engine latency when persisted samples exist. Network mode performs real requests and should not be used as a high-frequency health check.

🌐 Proxy Troubleshooting & Upstream Rate Limiting

When configuring or diagnosing proxy connectivity, refer to these common edge cases:

1. Startup log shows 🌐 Proxy: disabled even though proxy.url is set?

   • Check useProxy flag: In ~/.my-websearch/config.json, setting "useProxy": false takes precedence and explicitly overrides and disables url (setting PROXY_URL='false'). Change "useProxy": true or simply omit the field (providing a valid url automatically enables the proxy).

   • System proxy is auto-detected: when no explicit proxy is configured, the OS system proxy (Windows registry / macOS scutil / HTTPS_PROXY) is used for overseas engines and generic fetches — you do not need PROXY_URL just because a proxy client is running. If detection fails or you want to pin a specific proxy, set PROXY_URL (e.g. http://127.0.0.1:7897).

   • Prefer ~/.my-websearch/config.json over client env: put proxy.url and API keys in the config file. Some MCP clients (e.g. WorkBuddy) fingerprint the server by command + args + env key names + cwd, so adding a PROXY_URL env entry changes the trust fingerprint and the server may silently drop out of the session. Config-file settings are read at startup and avoid that.

   • Where to put the proxy (the error message used to say only "MCP client config", which was ambiguous):

     Location	Use it when
     ~/.my-websearch/config.json → "proxy": { "url": "http://127.0.0.1:7897" }	Recommended. Applies to every client (CLI, daemon, MCP) and does not change client trust fingerprints.
     MCP client env → PROXY_URL	Per-client override. Restart the client; some clients include env keys in their trust fingerprint.
     Proxy client's "system proxy" switch	No config at all. Since 1.3.8 it is auto-detected for overseas engines and generic fetches.

2. Proxy works, but an overseas engine still fails or times out? (Per-engine diagnosis, not a config error)

   • DuckDuckGo returns HTTP 202 / challenge: the 202 body is DDG's anomaly CAPTCHA (human verification), not a plain per-IP rate limit; the d.js arithmetic solver does not bypass it. The engine first requests the HTML endpoint with a wreq Chrome-profile GET, which normally returns 200 even when the axios POST path is challenged with 202 — so a 202 now only appears if that path fails and the axios fallback is used. If you still see 202, confirm your build includes the wreq GET path (1.3.8 and earlier do not), then retry or check that the proxy is attached. DDG remains best-effort: the same exit can still be challenged intermittently, so do not treat it as an SLA; for stability use Bing/Startpage or configure BRAVE_API_KEY/EXA_API_KEY. Note: SEARCH_MODE=playwright:duckduckgo is not supported (DDG has no browser-rendering path), so it falls back to HTTP.

   • Bing returns HTTP 200 but 0 results: the wreq impersonation path now falls back to the axios SERP path automatically when it parses an empty page. If both paths stay empty, the query/IP is likely rate-limited.

   • Brave returns HTTP 429 Too Many Requests: after a limited number of requests the CloudFront edge starts serving an anti-bot challenge page (HTTP 429) to the exit IP, while the first requests often still depend on the client TLS/HTTP stack. Fresh-node measurements: axios (Node OpenSSL, HTTP/1.1) → 0/10, all 429; curl (Schannel, HTTP/1.1) → 7/10 then 3/3 429; wreq (HTTP/2 Chrome) → 429; .NET Schannel HTTP/2 → 200/429 mixed. After curl hit 429, retrying .NET Schannel HTTP/2 on the same URL was 0/3, and on fresh URLs 0/3 — once the IP quota is exhausted, changing the client stack does not help. The Brave web path always uses curl (Windows Schannel; no axios fallback) and trips a rolling 60-second window of 6 requests — the 7th request is challenged (measured on a clean exit at 1 request / 10 s: 6 × 200, then 429). The 429 body is not a plain quota page: it is Brave's SPA challenge (captcha + Argon2 challengeSet), so it is reported as bot_challenge rather than rate_limited. Recovery follows the window, not a fixed timer: requests pass again once the oldest request of the burst ages out of 60 s (measured 62.9 s after a 6-request burst; ~15 s when the same 6 requests were spread over 52 s). Retrying inside the window adds requests and extends it — probing every 15 s / 30 s kept the IP blocked for 255 s / 6.5 min respectively. The engine therefore trips the Brave circuit for 60 s (then 120 → 240 → 300 s) and retries once through a disguised desktop headless CDP browser that clicks the PoW 验证 button automatically (8/8 success, ~4–6 s per query on a hard-blocked IP). That fallback only serves that search and does not clear the HTTP window: after a successful browser pass, the next HTTP attempt succeeded once and then hit 429 again. Client-stack swaps do not help either (axios 0/10; wreq HTTP/2 429; curl Schannel HTTP/1.1 7/10 before the window trips). The official API (BRAVE_API_KEY) remains the stable path. BRAVE_BROWSER_FALLBACK=0 disables both the retry and the window. After a 429 the engine enters a 180 s browser-only window (long-running MCP/daemon processes only; one-shot CLI invocations start fresh): HTTP is skipped entirely and every Brave call goes straight to the disguised browser (no wasted curl request that would only extend the server-side window); browser-only engines stay eligible but are ranked last by auto routing and cascade, so they do not slow down normal searches; if the browser fails too, the circuit opens (60 → 120 → 240 → 300 s) and routing/cascade substitute other engines.

   • Startpage times out or Anubis PoW difficulty spikes: difficulty 6 is now solved inline by up to 4 worker threads (~3–12s); >6 is skipped. Pass-challenge socket errors / transient 5xx are retried automatically.

   • 💡 Resolution: these are upstream anti-bot/rate-limit behaviours, not a broken MCP configuration. Handle them per engine: DDG 202 → upgrade/retry (the wreq GET path; in auto mode Yahoo also replaces an unhealthy DDG from the fallback pool); Brave 429 → the disguised-browser fallback usually rescues it (see above); otherwise wait out the window, rotate the exit IP, or set BRAVE_API_KEY; Bing 0 results → retry or adjust the query; Startpage PoW → retry. Switch exit IP only after the engine-specific step fails. As a last resort use engines: ["bing", "baidu"], or configure an official key (BRAVE_API_KEY / EXA_API_KEY) for an SLA path.

🔐 Security Model (read before exposing the server)

The defaults are safe as a package, and every one of them matters:

Relaxing one without the others silently removes the protection:

• HOST=0.0.0.0 (Docker) → the Host allowlist follows the bind address automatically; if you also use a public domain, add it via OPEN_WEBSEARCH_ALLOWED_HOSTS (comma-separated).

• Exposing to the public internet → put it behind a reverse proxy with authentication (or an SSH tunnel). The daemon prints a warning when bound to a non-loopback address.

• MCP session IDs are identifiers, not credentials — anyone who can reach the port can resume a session.

• FETCH_WEB_INSECURE_TLS=true disables certificate verification for user-supplied URLs; only enable it for a trusted network.

• The app copies API keys it finds in ~/.gemini/config/mcp_config.json / ~/.workbuddy/mcp.json into ~/.my-websearch/config.json (mode 0600) on first run. Run my-websearch doctor to see where each key came from; set MYWEBSEARCH_NO_AUTO_BOOTSTRAP=1 to disable the copy.

⚙️ Configuration & Environment Variables

Global Configuration (~/.my-websearch/config.json)

To share settings across all MCP clients without duplicate configuration:

{
  "apiKeys": {
    "context7": "ctx7sk-...",
    "exa": "...",
    "github": "ghp_...",
    "gitee": "...",
    "brave": "..."
  },
  "proxy": {
    "url": "http://127.0.0.1:7897",
    "useProxy": true,
    "engines": ["duckduckgo", "exa", "brave", "startpage"]
  }
}

⚠️ Note on useProxy: When "url" is configured, proxy is enabled automatically. Setting "useProxy": false will explicitly override and disable url. If you want proxy enabled, set "useProxy": true or omit the field entirely. Keys & restart: entries under apiKeys are mapped to env vars at startup (github → GITHUB_TOKEN, gitee → GITEE_TOKEN, context7 → CONTEXT7_API_KEY, …) and only fill variables that are not already set in the client env. Restart the MCP server after editing the file; my-websearch doctor shows which keys are currently visible.

📋 Exhaustive Environment Variables Reference

Can be set via the global configuration file (~/.my-websearch/config.json), local .env, or system environment variables:

Precedence (highest first): process environment → project .env → project .my-websearch.json → ~/.my-websearch/config.json → built-in defaults.

0. 🖥️ Server & Runtime

1. 🔍 Search & Routing Orchestration

2. 🌐 Proxy & Network Security

Custom proxy engine lists: if you set proxy.engines in ~/.my-websearch/config.json or use the ;engine1,engine2 suffix of PROXY_URL, yahoo and bing-news are added automatically because they cannot be reached directly from CN (Yahoo times out in 15–20s; Bing News returns a 302/HTML page). Removing the list entirely is still recommended: defaults include them and keep the direct API engines (GitHub / Stack Overflow / Hacker News / arXiv) off the proxy.

3. 🔐 Credentials & API Keys

4. 🖥️ Server Modes, Concurrency & Session Reaper

5. 🧭 CDP Browser & Stealth Controls (legacy Playwright env names kept)

Two independent knobs — don't mix them up:

• BROWSER_DRIVER picks the driver library: cdp (default, built in via chrome-remote-interface) or playwright (legacy escape hatch that requires you to install playwright-core).

• PLAYWRIGHT_CDP_ENDPOINT / CHROME_PATH / BROWSER_PATH / PLAYWRIGHT_EXECUTABLE_PATH decide where the browser comes from: an already-running local/remote CDP endpoint, an explicit executable, or auto-detection.

Every browser path — Bing fallback, Startpage, DuckDuckGo, Brave, fetchWebContent, CSDN article fetch, and doctor --browser — goes through the same entry point, so these settings apply to all of them.

6. 🛠️ Application Directory & Tool Customization

💡 Zero-Config Upgrades (Redundant Variables Removed):

• CORS Switches (ENABLE_CORS/CORS_ORIGIN): CORS is disabled by default; enable it explicitly and restrict origins when browser clients need cross-origin access.

• Browser Backend: raw CDP via chrome-remote-interface; playwright-core is no longer bundled. PLAYWRIGHT_* names and SEARCH_MODE=playwright remain as compatibility aliases.

• Emoji Switch (OPEN_WEBSEARCH_NO_EMOJI): Automatic console code page detection smoothly falls back to ASCII tags on legacy terminals.

• Fragmented Debug Flag (OPEN_WEBSEARCH_DEBUG): Unified into standard LOG_LEVEL=debug.

• Cookie Persistence TTL (MYWEBSEARCH_COOKIE_TTL_DAYS): Solidified to 30 days with automated sliding renewal.

🤝 Contributing & Tests

Pull requests and issues are welcome! To run the test suite locally:

npm install
npm run build
npm test              # Run the Vitest suite (`npm test` prints the current file/test counts)
npm run check:docs    # Verify README env tables match docs/env-schema.json (edit the schema, then `npm run docs:env`)
npm run test:live     # Run bounded-concurrency live integration tests
npm run eval:live     # Run 23-case live STDIO JSON-RPC evaluation suite

Acknowledgements

Author: wtznicy

This project evolved from Open-WebSearch (originally created by Aas-ee) — special thanks to the original author. Thanks also to:

• wreq-js for native Chrome TLS/HTTP2 fingerprint impersonation

• Context7 for powering official documentation retrieval

• Mozilla Readability & Turndown for content extraction and GFM Markdown formatting