# Yet Another WebFetch MCP Alternative

[English](#english) | [中文](#中文)

---

## English

A local MCP server that fetches any http/https webpage, extracts the main content, and converts it to Markdown. Designed as a drop-in replacement for Claude Code's built-in `WebFetch` tool, which may fail under certain network / proxy environments (e.g., "Unable to verify if domain is safe").

### Features

- **Content extraction** — Mozilla Readability for article pages; falls back to full-page conversion for non-article content.
- **HTML → Markdown** — powered by turndown; preserves headings and code blocks.
- **Auto proxy detection** — picks up Windows / macOS / Linux system proxy automatically (env vars also supported).
- **Truncation** — defaults to 50,000 characters to prevent context overflow.
- **20s timeout**, redirect-follow, browser User-Agent.
- **SSRF protection** — http/https only; blocks localhost and private IP ranges (IPv4 & IPv6).

### Installation

Requires Node.js ≥ 20.

```bash
git clone <repo-url> Yet-Another-WebFetch-MCP-Alternative
cd Yet-Another-WebFetch-MCP-Alternative
npm install
```

### Setup with Claude Code

Add the following to `~/.claude.json` under `mcpServers` (user-level) or to a project-local `.mcp.json`:

```json
{
  "mcpServers": {
    "webfetch": {
      "type": "stdio",
      "command": "node",
      "args": ["<path-to-repo>\\index.js"]
    }
  }
}
```

> **Note:** On Windows, avoid using `npx` as the `command` — it silently fails in non-TTY environments.

Reload the window (`Ctrl+Shift+P` → **Developer: Reload Window**), then type `/mcp` in a conversation to verify the connection.

### Usage

The tool is registered as `fetch_web`.

| Parameter | Required | Description |
|---|---|---|
| `url` | Yes | Full URL, http or https |
| `maxChars` | No | Max characters returned, default `50000` |
| `timeout` | No | Timeout in milliseconds, default `20000` |

Claude will invoke the tool automatically when it needs to fetch a webpage — no manual prompting required.

### Proxy

The server auto-detects the system proxy on all three platforms:

| Platform | Detection method |
|---|---|
| **Windows** | Registry `Internet Settings\ProxyServer` |
| **macOS** | `scutil --proxy` |
| **Linux** | GNOME `gsettings` |

To override manually, set the `HTTPS_PROXY` or `HTTP_PROXY` environment variable in the MCP config:

```json
{
  "webfetch": {
    "type": "stdio",
    "command": "node",
    "args": ["<path-to-repo>\\index.js"],
    "env": {
      "HTTPS_PROXY": "http://127.0.0.1:7890"
    }
  }
}
```

### Testing

```bash
npm test          # Handshake & tool list test (no network required)
LIVE=1 npm test   # End-to-end fetch against docs.tavily.com
```

### Known Limitations

- No JavaScript rendering — dynamic pages only return the initial HTML (same as most scraping tools).
- Non-UTF-8 pages may produce garbled output in edge cases.
- Sites with strict anti-bot measures may return error pages or block requests.
- Domains unreachable from the local network (e.g., DNS-blocked in certain regions) cannot be fetched.

### License

MIT

---

## 中文

一款本地 MCP 服务器：抓取任意 http/https 网页，提取正文并转换为 Markdown。作为 Claude Code 内置 `WebFetch` 工具的替代——内置 WebFetch 在部分网络 / 代理环境下会报 "Unable to verify if domain is safe" 而不可用。

### 特性

- **正文提取** — 基于 Mozilla Readability，非文章页面自动回退全文转换。
- **HTML → Markdown** — 基于 turndown，保留标题与代码块。
- **自动代理检测** — 自动读取 Windows / macOS / Linux 系统代理，也支持环境变量覆盖。
- **截断** — 默认 50000 字符，防止上下文溢出。
- **20s 超时**、重定向跟随、浏览器 UA。
- **SSRF 防护** — 仅 http/https，拒绝本机与内网地址（含 IPv6）。

### 安装

需要 Node.js ≥ 20。

```bash
git clone <仓库地址> Yet-Another-WebFetch-MCP-Alternative
cd Yet-Another-WebFetch-MCP-Alternative
npm install
```

### 接入 Claude Code

在 `~/.claude.json` 的 `mcpServers`（用户全局）或项目根目录 `.mcp.json` 中加入：

```json
{
  "mcpServers": {
    "webfetch": {
      "type": "stdio",
      "command": "node",
      "args": ["<仓库路径>\\index.js"]
    }
  }
}
```

> **注意：** Windows 下不要用 `npx` 作为 `command`（无 TTY 环境下会静默失败）。

保存后重载窗口（`Ctrl+Shift+P` → **Developer: Reload Window**），在对话中输入 `/mcp` 确认连接状态。

### 使用

工具名为 `fetch_web`。

| 参数 | 必填 | 说明 |
|---|---|---|
| `url` | 是 | 完整 URL，http 或 https |
| `maxChars` | 否 | 返回内容最大字符数，默认 `50000` |
| `timeout` | 否 | 超时毫秒数，默认 `20000` |

Claude 会在需要抓取网页时自动调用，无需手动引导。

### 代理

服务器自动检测三平台系统代理：

| 平台 | 检测方式 |
|---|---|
| **Windows** | 注册表 `Internet Settings\ProxyServer` |
| **macOS** | `scutil --proxy` |
| **Linux** | GNOME `gsettings` |

如需手动覆盖，在 MCP 配置中设置 `HTTPS_PROXY` 或 `HTTP_PROXY` 环境变量：

```json
{
  "webfetch": {
    "type": "stdio",
    "command": "node",
    "args": ["<仓库路径>\\index.js"],
    "env": {
      "HTTPS_PROXY": "http://127.0.0.1:7890"
    }
  }
}
```

### 测试

```bash
npm test          # 握手与工具清单测试（不依赖网络）
LIVE=1 npm test   # 额外做一次端到端抓取（docs.tavily.com）
```

### 已知限制

- 不渲染 JavaScript — 动态页面只能拿到初始 HTML（与多数抓取工具一致）。
- 非 UTF-8 编码页面在极端情况下可能乱码。
- 严格反爬站点可能被拦截或返回错误页。
- 本机网络 DNS 不可达的域名无法抓取。

### 许可证

MIT