Skip to main content
Glama
guanyuyan

scrape-mcp

by guanyuyan
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

scrape-mcp

⚠️ 这是 scrape-mcp 的官方源仓库（upstream）。 若在别处看到同名/相似项目并带不同作者署名，即为转载或派生，非原作者发布。

反爬感知抓取 + 极省 Token 内容精简的 MCP Server —— 让 AI 读真实网页时，只花 1%~5% 的 token 就能拿到干净、真实、免登录的正文。

自动化分三层应对：默认 curl_cffi（伪装 TLS/JA3 + HTTP2）低成本打头阵，命中 Cloudflare/验证码/WAF 才升级 Playwright 真浏览器渲染；遇到"登录才给内容"的站，先弹窗手动登录一次、登录态落盘，之后自动带上、过期会提示重登。

License: MIT Release CI

功能说明

它解决什么：模型读网页不是在读正文，而是在读整棵 DOM（脚本、导航、版权、链接一应尽收），又贵又脏。这个工具把页面剪成"只有正文"再喂给 AI。

核心价值（三点）

  1. 极省 token —— 属性剥离 + 模板块丢弃 + 站外链接降级 + 表格转 CSV + 预算截断。实测 MDN 整页 214KB → 数百 token，压缩率约 1%。

  2. 自动分级反爬 —— 85% 的站只走最轻的 L1（快、并发高），拦截/JS 骨架页才升级 L2 真浏览器 + stealth 伪装，绝不暴力逐站上浏览器。

  3. 登录墙处理 + 如实上报 —— 判定登录/拦截时返回 login_required/blocked，引导登录而非硬撞，从不让模型把验证码当正文。

怎么用（一行）

web_fetch(url="https://xxx/article/1") → JSON：{ok, content(极省正文), token_estimate, tier, blocked?, login_required?, cached?}

覆盖的站点类型：文章页 / 文档站 / 门户与列表页 / 需要 JS 渲染的 SPA / 需要登录才给正文的站（如知乎）。

形态：本地 MCP Server（stdio/HTTP 均可），隐私可控；支持单页 web_fetch、批量 web_batch、登录持久化 login，缓存默认开启。

Related MCP server: SteadyFetch

特性

  • 分级抓取：L1 curl_cffi → L2 Playwright，拦截才升级，不误伤正常页面。

  • 内容密度选根：从 article/main/div 中挑出"恰好装住正文的最紧容器"，避开导航/侧栏。

  • compact 精简：属性剥离、模板块丢弃、站外链接降级、表格转 CSV、token 预算截断， 压缩率可达 1~20% 数量级（如 MDN 214KB → 数百 token）。

  • 拦截如实上报：不把挑战页/空壳页当正文喂给模型，返回 blocked/block_reason。

  • 登录态持久化：login 工具手动登录并落盘，web_fetch 自动带上；过期会提示重登。

  • 缓存与批量：L1 明文响应按 TTL 磁盘缓存；web_batch 并发抓取一批 URL。

  • Stealth 加固：Wipe webdriver、UA 数据、Canvas/WebGL 确定性指纹噪声、时区对齐。

安装

# L2 渲染所需浏览器（国内可用镜像）
$env:PLAYWRIGHT_DOWNLOAD_HOST="https://cdn.npmmirror.com/binaries/playwright"
pip install "playwright>=1.49"
playwright install chromium

# 或复用系统浏览器，无需下载
$env:SCRAPE_MCP_L2_CHANNEL="chrome"   # 或 msedge

启动（MCP，stdio）

scrape-mcp
# SSL_DIR 切换协议：stdio（默认）

环境变量（前缀 SCRAPE_MCP_）

全部配置可用环境变量（SCRAPE_MCP_ 前缀）设置，也可集中放到项目根目录 .env 文件里。

# 项目根目录执行
cp .env.example .env    # 生成配置模板
# 编辑 .env 后无需改代码，直接启动即可

优先级：实际进程环境变量 > .env > 默认值。.env 被 gitignore，不入库；.env.example 是随仓库提交的模板。

变量

默认

说明

IMPERSONATE

chrome

curl_cffi 指纹模板

REQUEST_TIMEOUT

20

L1 超时（秒）

PROXY

—

代理，L1/L2 共用

L2_ENABLED

true

是否启用 Playwright 升级链

L2_CHANNEL

—

关联浏览器：空=自带 Chromium，chrome/msedge=系统

L2_SETTLE_TIMEOUT

8

挑战页等待窗口（秒）

L2_TIMEZONE

Asia/Shanghai

隐身加固时区

DATA_DIR

~/.scrape_mcp

登录态/缓存放盘目录

CACHE_TTL

300

L1 缓存 TTL（秒），0 关闭

BATCH_MAX_CONCURRENCY

4

web_batch 并发上限

BATCH_MAX_URLS

20

web_batch 单批上限

RESPECT_ROBOTS

true

默认遵循目标站 robots.txt；false 忽略（仅建议自用合规场景关闭）

ALLOWED_HOSTS

—

主机白名单（逗号分隔）；非空时仅允许名单内主机

DENIED_HOSTS

—

主机黑名单（逗号分隔），优先于白名单

MAX_QPS

10

全网抓取限流（每秒请求数）；0 不限速

HTTP 模式（Postman / curl 直连）

默认 stdio，改环境变量即可切到 HTTP，无需改代码：

$env:SCRAPE_MCP_TRANSPORT="streamable-http"
$env:SCRAPE_MCP_HTTP_PORT="8000"
python -m scrape_mcp.server

以 json_response=1（默认）启动，请求/响应都是普通 JSON（非事件流）。stateless 模式无需先 initialize，直接发 tools/call 即可。

必需项：请求头 MCP-Method、Mcp-Name；body 内 params 需带 _meta（protocolVersion + clientCapabilities）。完整示例见下。

MCP 工具

  • web_fetch(url, max_tokens=4000, link_policy="internal") → 状态 + 极省正文

    • link_policy: internal（站内相对链接+站外降级）/ all / none

  • web_extract(url, schema={...}) → 按字段 schema 抽取结构化 JSON（可入库）

    • 字段类型：text（默认）/ attr（需配 attr 取属性）/ count / list（list_key 取值）

  • web_batch(urls, max_tokens=3000, link_policy="internal") → 并发抓取一批，逐 URL 返回；单条异常不影响整批

  • login(url, timeout=180) → 打开带界面浏览器手动登录并持久化登录态

返回为 JSON 字符串（落在 content[0].text，请解析文本，勿依赖 structured_content）。

web_extract 示例

schema='{"fields": {
  "title":    {"selector": "h1", "type": "text"},
  "main_link":{"selector": "a", "type": "attr", "attr": "href"},
  "tag_count":{"selector": ".tag", "type": "count"},
  "tags":     {"selector": ".tag", "type": "list", "list_key": "text"}
}}'
# → {"ok":true, "data": {"title":"…","main_link":"…","tag_count":3,"tags":["a","b","c"]}, "missing":[]}

目录结构

src/scrape_mcp
├── server.py          # MCP 入口 + 工具（web_fetch/web_batch/login）
├── config.py          # 环境变量配置
├── tokenizer.py       # tiktoken 计数 + 预算截断
├── core/
│   ├── http_client.py # L1 curl_cffi + 磁盘缓存
│   ├── fetcher.py     # 分级调度（L1→L2），对工具层返回 FetchOutcome
│   ├── detector.py    # 拦截/骨架页判定
│   ├── browser.py     # L2 Playwright + stealth
│   ├── session.py     # 登录态持久化
│   └── cache.py       # HTTP 响应缓存
└── extract/
    └── compact.py     # DOM 剪枝 → 极省 token 正文

开发

pip install -e ".[dev]"
python -m pytest

Available Tools

3 tools
loginA

打开带界面的浏览器手动登录，并把该站登录态持久化到磁盘。

用于 web_fetch 报告 login_required=true 的站点：浏览器会以非无头方式打开目标页， 请在弹出的窗口里完成登录；检测到登录成功或超时后，登录态（cookie 等）会被保存， 之后该站的 web_fetch 会自动带上登录态。

Args: url: 目标站点任一页面 URL（按域名区分登录态）。 timeout: 等待手动登录完成的秒数，超时也会保存当前状态便于下次继续。

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
timeoutNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description fully carries the behavioral burden. It discloses that the browser opens non-headless, that login must be manually completed, that cookies are persisted per domain, and that timeout still saves current state—covering the key behavioral side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a short overview followed by an Args section. It is slightly verbose in places, but every part contributes useful operational context, and the key workflow information is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the manual, interactive nature of this tool, the description covers the full call flow: why to invoke it, what the user will see, how the state is saved, and what happens after timeout. The existing output schema covers return values, so no additional explanation is needed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description fully documents both parameters: url specifies any page URL of the target site with login state keyed by domain, and timeout sets the manual-login waiting period with a defined timeout behavior. This adds substantial meaning beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: opening a GUI browser for manual login and persisting the site's login state to disk. It also explicitly ties itself to the web_fetch login_required=true workflow, which clearly distinguishes it from the sibling fetch tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to use this tool: when web_fetch reports login_required=true for a site. It also explains the follow-up behavior (web_fetch will automatically carry the saved login state afterward), giving the agent clear situational guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_batchA

并发抓取一批 URL，各自走完整分级链路，返回逐个结果。

Args: urls: 目标 URL 列表（最多 batch_max_urls 个，默认 20）。 max_tokens: 每个 URL 的正文 token 预算上限。 link_policy: 同 web_fetch。

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYes
max_tokensNo
link_policyNointernal

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden and does add useful behavior: concurrency, full processing chain per URL, per-URL token limits, and a batch default of 20. It does not cover failure handling, rate limits, or clarify what '完整分级链路' actually entails.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact, front-loaded with the core behavior, and uses a clear argument list with no filler. Each sentence contributes meaningful information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a three-parameter tool with an output schema present, the description covers the essentials: target URLs, per-URL token budget, link policy, and result granularity. It is incomplete in that it omits error semantics, behavior when the batch limit is exceeded, and the precise meaning of the full processing chain.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description compensates well by defining all three parameters: urls with a maximum/default count, max_tokens as a per-URL budget, and link_policy as equivalent to web_fetch. It loses one point because 'batch_max_urls' is referenced but not fully explained and link_policy values are not expanded.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: '并发抓取一批 URL' (concurrently fetch a batch of URLs) and notes that results are returned per URL. This is clear and the batch name distinguishes it from web_fetch, though it does not explicitly contrast the two siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The batch behavior implies use for multiple URLs versus web_fetch for a single URL, and link_policy is explicitly delegated to web_fetch semantics. However, it does not state when to prefer this tool over web_fetch or mention alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_fetchA

抓取网页并返回极省 token 的紧凑正文（自动辨别反爬拦截页并如实上报）。

Args: url: 目标网页完整 URL，需带 http/https 协议头。 max_tokens: 正文 token 预算上限，超出时保留头尾、省略中段。 link_policy: 链接处理策略。internal=站内链接保留为相对路径、站外降级为纯文本（默认，最省）； all=站内外链接全保留；none=全部降级为纯文本。

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
max_tokensNo
link_policyNointernal

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and does a good job: it discloses anti-crawler interception detection, truthful reporting, and the truncation strategy of keeping head/tail while omitting the middle when max_tokens is exceeded. This goes beyond what the schema alone would tell an agent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact, front-loaded with the main purpose, and uses a clear bulleted Args section. Every sentence adds useful information without redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-value shape does not need to be described. The description covers the operation, URL requirement, token budget behavior, link policy variants, and an important edge case (anti-crawling pages). It is slightly incomplete only in not addressing relationship to sibling tools such as web_batch and login.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description fully compensates by explaining the URL protocol requirement, the max_tokens truncation behavior, and every link_policy option with its default and trade-off. This gives an agent actionable parameter-level semantics that the raw schema lacks.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear verb and resource: it fetches a web page and returns a token-efficient compact body. It also signals a distinctive behavior (auto-detecting anti-crawling block pages), which helps distinguish it from generic fetch tools, though it does not explicitly contrast with its siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains what the tool does and how parameters work, but gives no guidance on when to prefer web_fetch over web_batch or login, nor any conditions where it should not be used. Usage context is only implied by the tool name and the single-page nature of the description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 3 tool updatesv0.1.0
    • First observedlogin
    • First observedweb_batch
    • First observedweb_fetch
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.1/5.0

Scored across 3 tools

Disambiguation5/5

web_fetch handles a single URL, web_batch handles multiple URLs concurrently, and login handles authentication persistence. These purposes are completely distinct, so an agent should never struggle to choose the right tool.

Naming Consistency4/5

web_fetch and web_batch share a clear web_ prefix and read as fetch/batch operations, which is consistent. login deviates from the prefix pattern, but its purpose is obvious and it is still a simple verb-style name.

Tool Count5/5

Three tools is a tight, focused set for a scraping server: single fetch, batch fetch, and authentication handling. There is no redundancy or bloat, and each tool earns its place.

Completeness4/5

The main scraping flows are covered: one-page fetches, multi-page fetches, and login-gated sites. Minor gaps exist, such as no way to explicitly manage or clear saved login sessions, but agents can still complete the core scraping lifecycle.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • cleanfetch

    belzorash
    A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to read web pages reliably, returning clean markdown content, hyperlinks, and metadata without navigation or ad noise.
    3
    6 npm
    MIT

  • SteadyFetch

    carsonlabs
    A
    license
    Not graded
    quality
    D
    maintenance
    Provides AI agents with reliable web fetching capabilities, handling retries, caching, and anti-bot bypass automatically.
    MIT

  • Web Content Extractor MCP

    agenson-tools
    A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to extract clean, structured web content (articles, tables, links, visual layouts) optimized for LLM token efficiency, with fast response times and optional JavaScript support.
    5
    29 npm
    MIT