🔍 web-search-cli

双引擎(Bing + DuckDuckGo)网页搜索与正文抓取工具

既能作为 MCP 服务器接入 AI Agent,也能作为独立命令行工具直接使用。

无需 API Key · 无速率限制 · 代理可选

中文 · English

✨ 简介

web-search-cli 直接解析 Bing 和 DuckDuckGo 的公开 HTML 搜索页,解码它们的跳转链接、跨引擎去重,返回干净的结构化结果。两种用法:

• 🧩 作为 MCP 服务器 —— 接入 Claude、Cline 或任意兼容 MCP 的客户端,给你的 Agent 配上不受限的联网搜索能力;

• 💻 作为命令行工具 —— 在终端里直接搜索、抓取网页正文。

Related MCP server: Web Search MCP Server

🎯 特性

📦 环境要求

• Node.js >= 18(依赖全局 fetch 与 undici)

🛠️ 安装

git clone https://github.com/AusertDream/web-search-cli.git
cd web-search-cli
npm install

想在任意目录直接用 web-search 命令,全局链接一下:

npm link

💻 命令行用法

# 搜索(默认:10 条结果、lang=en、engine=both)
web-search search "rust async runtime comparison"

# 更多结果 + 中文 + 仅用 Bing
web-search search "开源大模型推理框架" 15 zh-CN bing

# 抓取并提取网页正文(默认上限 8000 字符)
web-search fetch "https://example.com/article" 5000

没执行 npm link 时直接调用脚本:

node cli.mjs search "你的查询"

参数顺序:

引擎 可选 bing / duckduckgo / both,默认 both。

🧩 作为 MCP 服务器使用

服务器通过 stdio 走 MCP 协议。在客户端的 MCP 配置中加入即可。以 Claude / Cline 的 mcp.json 为例:

{
  "mcpServers": {
    "web-search": {
      "command": "node",
      "args": ["/绝对路径/web-search-cli/server.js"]
    }
  }
}

需要走代理时,补一个 env 块:

{
  "mcpServers": {
    "web-search": {
      "command": "node",
      "args": ["/绝对路径/web-search-cli/server.js"],
      "env": {
        "HTTPS_PROXY": "http://127.0.0.1:7890",
        "HTTP_PROXY": "http://127.0.0.1:7890"
      }
    }
  }
}

🧰 提供的工具

web_search —— 网页搜索

fetch_page —— 抓取并提取网页正文

⚙️ 工作原理

• Bing:解析 bing.com/search 中的 li.b_algo 区块,并把 base64 编码的 ck/a?u= 跳转链接还原成真实 URL。

• DuckDuckGo:解析 html.duckduckgo.com/html/ 中的 .result 区块,解开 uddg= 跳转参数,并跳过广告结果。

• both:合并两个引擎的结果,再按 域名 + 标题 去重。

⚠️ 由于依赖搜索引擎的公开 HTML 结构,若它们改版,解析逻辑可能需要相应更新。

🌐 关于代理

代理由环境变量控制:设置 HTTPS_PROXY 或 HTTP_PROXY 即启用,不设则直连。代码中不硬编码任何代理地址。

🇨🇳 国内用户请注意:必须配置代理。 Bing 与 DuckDuckGo 在中国大陆网络环境下通常无法直接访问(DuckDuckGo 被屏蔽,Bing 国际版亦不稳定),不挂代理会连接超时、搜不到结果。请务必先设置代理再使用:

export HTTPS_PROXY=http://127.0.0.1:7890   # 改成你自己的代理地址端口
export HTTP_PROXY=http://127.0.0.1:7890

作为 MCP 服务器时,把代理写进 mcp.json 的 env 块即可(见上文 MCP 配置示例)。

🤝 贡献 / Contributing

欢迎 Issue 与 PR。引擎改版导致解析失效时,修复解析器的 PR 尤其受欢迎。

📄 许可证

MIT © AusertDream

🔍 web-search-cli  (English)

🌟 Introduction

A dual-engine (Bing + DuckDuckGo) web search and page-fetch tool. It scrapes the public HTML search pages of Bing and DuckDuckGo, decodes their redirect URLs, deduplicates across engines, and returns clean structured results. Two ways to use it:

• 🧩 As an MCP server — plug it into Claude, Cline, or any MCP-compatible client to give your agent unrestricted web search;

• 💻 As a standalone CLI — run searches and extract page text right from your terminal.

No API keys required.

🎯 Features

📦 Requirements

• Node.js >= 18 (uses the global fetch and undici).

🛠️ Install

git clone https://github.com/AusertDream/web-search-cli.git
cd web-search-cli
npm install

To use the web-search command anywhere, link it globally:

npm link

💻 CLI usage

# Search (defaults: 10 results, lang=en, engine=both)
web-search search "rust async runtime comparison"

# More results, Chinese, Bing only
web-search search "开源大模型推理框架" 15 zh-CN bing

# Fetch and extract a page's main text (default max 8000 chars)
web-search fetch "https://example.com/article" 5000

Without npm link, call the script directly:

node cli.mjs search "your query"

Argument order:

engine is one of bing / duckduckgo / both (default both).

🧩 MCP server usage

The server speaks MCP over stdio. Add it to your client's MCP config. Example for Claude / Cline (mcp.json):

{
  "mcpServers": {
    "web-search": {
      "command": "node",
      "args": ["/absolute/path/to/web-search-cli/server.js"]
    }
  }
}

To route through a proxy, add an env block:

{
  "mcpServers": {
    "web-search": {
      "command": "node",
      "args": ["/absolute/path/to/web-search-cli/server.js"],
      "env": {
        "HTTPS_PROXY": "http://127.0.0.1:7890",
        "HTTP_PROXY": "http://127.0.0.1:7890"
      }
    }
  }
}

🧰 Exposed tools

web_search — search the web.

fetch_page — fetch and extract readable text from a URL.

⚙️ How it works

• Bing: parses li.b_algo blocks from bing.com/search and decodes the base64 ck/a?u= redirect wrappers back to real URLs.

• DuckDuckGo: parses .result blocks from html.duckduckgo.com/html/ and unwraps the uddg= redirect params, skipping ad results.

• both: concatenates results from both engines, then deduplicates by domain + title.

⚠️ Because this relies on the engines' public HTML markup, result parsing may need updating if they change their page structure.

🌐 Proxy

Proxy is controlled entirely by environment variables. Set HTTPS_PROXY or HTTP_PROXY to enable it; leave them unset for direct connections. No proxy address is hardcoded.

🇨🇳 Users in mainland China: a proxy is required. Bing and DuckDuckGo are generally not reachable from within mainland China (DuckDuckGo is blocked, and international Bing is unstable), so without a proxy you'll hit connection timeouts and get no results. Set a proxy first:

export HTTPS_PROXY=http://127.0.0.1:7890   # replace with your own proxy host:port
export HTTP_PROXY=http://127.0.0.1:7890

When running as an MCP server, put the proxy in the env block of mcp.json (see the MCP config example above).

🤝 Contributing

Issues and PRs welcome — especially parser fixes when an engine changes its markup.

📄 License

MIT © AusertDream