Skip to main content
Glama
bpzx

web-fetch-mcp

by bpzx
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

web-fetch-mcp

一个用于网页抓取和搜索的 MCP(Model Context Protocol)服务器。

  • web_fetch —— 把网页正文提取为 Markdown(或纯文本)。以 Tavily Extract 为主用,失败或返回空时自动回退到 Firecrawl Scrape。

  • web_search —— 用 Tavily Search 搜索网页(返回标题、URL、摘要,可选合成答案)。无备用。

API key 从环境变量读取。未配置 key 的 provider 会被直接跳过,不会向它发起任何请求。

适用场景

  • 抓取单个网页的正文内容(博客、文档、新闻等)。

  • 当主用服务(Tavily)不可用时,自动用备用服务(Firecrawl)兜底,提高可用性。

  • 用 Tavily 做联网搜索,获取最新信息。

Related MCP server: webcrawl-mcp

环境要求

  • Python ≥ 3.11

  • Tavily API key(TAVILY_API_KEY)

  • 可选:Firecrawl API key(FIRECRAWL_API_KEY),用于抓取回退。不配则 web_fetch 仅用 Tavily。

运行方式

本服务用 Python 实现,推荐用 uvx(uv 的命令运行器)运行 —— 无需全局安装、自动隔离环境。

方式一:uvx(推荐)

无需 pip install,直接运行。两种来源:

# 已发布到 PyPI 时
uvx web-fetch-mcp

# 未发布时,直接从 git 仓库运行(把 bpzx/web-fetch-mcp 换成你的仓库地址)
uvx --from "git+https://github.com/bpzx/web-fetch-mcp" web-fetch-mcp

uvx 会根据 pyproject.toml 的 [project.scripts] 入口点(web-fetch-mcp = web_fetch_mcp.server:main)找到并运行服务。首次运行会自动创建隔离环境并安装依赖。

方式二:本地 pip 安装

在本目录下:

pip install -e .

然后即可通过命令 web-fetch-mcp 或 python -m web_fetch_mcp 启动。

调试 / 检查

用官方 MCP Inspector 可视化调试:

npx @modelcontextprotocol/inspector uvx web-fetch-mcp

配置 API key

在环境(或 MCP 客户端加载的 .env)中设置:

export TAVILY_API_KEY=tvly-xxxxxxxxxxxx
export FIRECRAWL_API_KEY=fc-xxxxxxxxxxxx   # 可选

仓库内附 .env.example。

出站代理(可选)

当 Tavily / Firecrawl 的 API 端点在你的网络下无法直连或不稳定(例如国内网络),可通过 WEB_FETCH_PROXY 指定本地代理:

export WEB_FETCH_PROXY=http://127.0.0.1:7890

设置 WEB_FETCH_PROXY 后它优先级最高,并禁用 httpx 的环境变量回退;未设置时,httpx 仍会读取标准的 HTTP_PROXY / HTTPS_PROXY / ALL_PROXY 环境变量。

工具

web_fetch(url, ...)

参数

类型

默认值

说明

url

str

必填

要抓取的页面 URL。

query

str | None

None

Tavily 会按该意图对提取内容重排。

format

"markdown" | "text"

"markdown"

输出格式。

extract_depth

"basic" | "advanced"

"basic"

Tavily 提取深度。

only_main_content

bool

True

仅 Firecrawl 生效 —— 去掉导航/广告/页脚。

timeout

int

30

单次请求超时,单位秒。

返回 { ok, url, provider, content, format, used_fallback, failed_providers, metadata }。 当结果由 Firecrawl 在 Tavily 失败后产出时,used_fallback 为 true。

web_search(query, ...)

参数

类型

默认值

说明

query

str

必填

搜索词。

max_results

int

5

结果数量(1–20)。

search_depth

"basic" | "advanced"

"basic"

advanced 更慢但更深入。

topic

"general" | "news"

"general"

切换为新闻搜索。

include_answer

bool

False

是否同时返回合成答案。

返回 { ok, query, answer, results: [{title, url, content, score}] }。

回退机制说明

web_fetch 按顺序遍历 provider 链:Tavily → Firecrawl。

  1. 若 Tavily 已配置且返回非空内容 → 直接返回。

  2. 若 Tavily 因可重试错误失败(auth、限流、5xx、网络、空内容)且 Firecrawl 已配置 → 尝试 Firecrawl,标记 used_fallback: true,并把 Tavily 的失败记录写入 failed_providers。

  3. 不可重试错误(如 404)会立即抛出,不尝试备用。

  4. 所有 provider 都失败或都未配置时,工具返回 ok: false 及结构化的 failures 列表,便于调用方处理。

未配置 key 的 provider 不会出现在链中,因此只配一个 key 即可实现 Tavily-only 或 Firecrawl-only。

客户端配置

ZCode(~/.zcode/mcp.json 或项目内 .zcode/mcp.json)

注意:ZCode 的配置文件格式与 Claude Desktop 不同 —— 顶层的 key 直接就是每个 server 的名字,没有 mcpServers 外层包裹,且每个 server 对象里需要带 "type": "stdio"。

{
  "web-fetch": {
    "type": "stdio",
    "command": "uvx",
    "args": ["web-fetch-mcp"],
    "env": {
      "TAVILY_API_KEY": "tvly-xxxxxxxxxxxx",
      "FIRECRAWL_API_KEY": "fc-xxxxxxxxxxxx",
      "WEB_FETCH_PROXY": "http://127.0.0.1:7890"
    }
  }
}

如果还没发布到 PyPI,改为从 git 运行(把 bpzx/web-fetch-mcp 换成你的仓库地址):

"args": ["--from", "git+https://github.com/bpzx/web-fetch-mcp", "web-fetch-mcp"]

Claude Desktop / Claude Code(claude_desktop_config.json)

{
  "mcpServers": {
    "web-fetch": {
      "command": "uvx",
      "args": ["web-fetch-mcp"],
      "env": {
        "TAVILY_API_KEY": "tvly-xxxxxxxxxxxx",
        "FIRECRAWL_API_KEY": "fc-xxxxxxxxxxxx",
        "WEB_FETCH_PROXY": "http://127.0.0.1:7890"
      }
    }
  }
}

通用 stdio 客户端

{
  "command": "uvx",
  "args": ["web-fetch-mcp"]
}

确保服务端继承的环境中已导出 TAVILY_API_KEY(以及可选的 FIRECRAWL_API_KEY / WEB_FETCH_PROXY)。

开发

pip install -e ".[dev]"
pytest

许可证

MIT

Available Tools

2 tools
web_fetchA

Fetch and extract the content of a web page as markdown (or plain text). Tries Tavily first; on failure or empty result, automatically falls back to Firecrawl. Requires TAVILY_API_KEY and/or FIRECRAWL_API_KEY in the server's environment.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
queryNo
formatNomarkdown
extract_depthNobasic
only_main_contentNo
timeoutNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description carries full burden. It discloses fallback behavior and API key requirements, but lacks details on error handling (e.g., if both services fail), rate limits, or handling of dynamic content. Adequate but not comprehensive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences: first states core purpose, second adds essential behavioral detail. No extraneous content, well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having an output schema (not shown), the description lacks crucial parameter explanations, error behavior, and auth specifics beyond key requirements. For a tool with 6 parameters and no schema descriptions, the description is too sparse to be complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% description coverage, and the tool description adds no information about any of the 6 parameters (url, query, format, extract_depth, only_main_content, timeout). The agent must rely solely on parameter names and defaults, which is insufficient for correct invocation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Fetch and extract' and the resource 'content of a web page', specifying output format (markdown/plain text). It also mentions fallback behavior, distinguishing it from sibling web_search which likely searches rather than fetches a specific URL.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides context for when to use by noting fallback to Firecrawl and required API keys. It doesn't explicitly state when not to use or compare to web_search, but the sibling context implies differentiation. A slight gap in explicit guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv0.1.0
    • First observedweb_fetch
    • First observedweb_search
Deploy Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.1/5.0

Scored across 2 tools

Disambiguation5/5

The two tools have distinct purposes: web_fetch retrieves content from a specific URL, while web_search performs a web search query. There is no overlap in functionality.

Naming Consistency5/5

Both tools follow the same verb_noun pattern with snake_case (web_fetch, web_search), making the naming predictable and consistent.

Tool Count4/5

With only 2 tools, the server is minimal but appropriate for its narrow scope of web fetching and searching. It could be expanded, but it's not overly thin for a focused utility.

Completeness4/5

The tool surface covers the basic needs of fetching a web page and searching the web. Minor gaps exist, such as no batch fetching or support for additional output formats, but no critical operations are missing.

Maintenance

ActivityMaintained
ResponsivenessSyncing

Resources

Related MCP Connectors

Related MCP Servers

  • metasearch-mcp

    YeonwooSung
    A
    license
    Not graded
    quality
    D
    maintenance
    MCP server for using various search tools like Tavily API. Planning to support various search tools (i.e. wiki search, searxng, etc)
    3
    MIT

  • webcrawl-mcp

    andyliszewski
    A
    license
    B
    quality
    C
    maintenance
    Lightweight MCP server for web scraping, search, and crawling. Uses local trafilatura/DuckDuckGo by default with optional Firecrawl fallback for transport-blocked pages.
    4
    15
    MIT

  • web-mcp-server

    RyanDam
    F
    license
    Not graded
    quality
    D
    maintenance
    MCP server that exposes web_search and web_fetch tools, allowing LLM applications to search the web via DuckDuckGo and fetch page content as cleaned markdown.
    -