セルフホスト型のModel Context Protocolサーバーで、LLMクライアント（Claude Code、Cursor、ChatGPT）に有料のスクレイピングサービスと同じツールサーフェス（scrape、crawl、map、search、extract、interact、deep_research）を提供し、すべて自前のハードウェア上で実行します。
有料のプロキシ/CAPTCHAサービスは不要です。アンチボットはセルフホスト（ヘッドレスChromium + playwright-stealth、robots.txt、ポライトなレート制限）です。防御の固いサイトではブロックされる可能性があります。制限事項を参照してください。
ツール | 機能 |
| 1つのURL → クリーンなMarkdown（ボイラープレート除去）。静的優先、JSページには自動でブラウザフォールバック。 |
| バックグラウンドのBFSクロールジョブ（重複排除、深さ/ページ上限）。結果をポーリング。 |
| ページ上のリンクを一覧表示（オプションで同一ドメインのみ）— 何をクロールするか決定。 |
| Web検索。プラグイン可能なバックエンド：DuckDuckGo（デフォルト）、SearXNG、Brave、またはTavily。 |
| ページを取得し、LLMを介してスキーマに一致する構造化JSONを抽出。 |
| トークン効率の良いARIAスナップショットを使用して、永続的なブラウザセッションを操作（クリック/入力/キー押下）。 |
| 検索 → 上位ソースを読む → 引用付きの総合レポートを返す。 |
extract と deep_research には ANTHROPIC_API_KEY が必要です。
Related MCP server: Fetch MCP Server
uv sync # install deps (uses the pinned uv.lock)
uv run playwright install chromium
export SCRAPER_AUTH_TOKEN=$(openssl rand -hex 32)
uv run web-scraper-mcp # HTTP server on http://127.0.0.1:8000/mcpStdio（ローカル、デスクトップクライアント用）：SCRAPER_TRANSPORT=stdio uv run web-scraper-mcp。
最も速い開始方法は、DockerHubからビルド済みイメージをプルすることです。
# Pull the latest image
docker pull PROG_UP_USERNAME/web-scraper-mcp:latest
# Run the container (with Anthropic / Claude)
docker run -p 8000:8000 \
-e SCRAPER_AUTH_TOKEN=your_secure_token_here \
-e ANTHROPIC_API_KEY=sk-ant-api03-... \
PROG_UP_USERNAME/web-scraper-mcp:latest
# Or run the container over stdio (useful for local MCP clients)
docker run -i --rm \
-e SCRAPER_TRANSPORT=stdio \
-e SCRAPER_AUTH_TOKEN=your_secure_token_here \
PROG_UP_USERNAME/web-scraper-mcp:latest（PROG_UP_USERNAME を実際のDockerHubユーザー名に置き換えてください）。
AnthropicのAPIの代わりにローカルでモデルを実行したい場合、サーバーは extract と deep_research ツールの代替バックエンドとしてOllamaを完全にサポートしています。
docker run -p 8000:8000 \
-e SCRAPER_AUTH_TOKEN=your_secure_token_here \
-e SCRAPER_OLLAMA_HOST=http://host.docker.internal:11434 \
-e SCRAPER_EXTRACT_MODEL=qwen3.5:2b \
-e SCRAPER_RESEARCH_MODEL=qwen3.5:2b \
PROG_UP_USERNAME/web-scraper-mcp:latest[!WARNING] コンテキストウィンドウは重要です！ Webスクレイピングは大量のMarkdownを生成します。
extractは最大100,000文字、deep_researchは最大16,000文字をLLMに送信できます。デフォルトでは、Claudeは巨大なコンテキストをネイティブに処理します。しかし、Ollamaのデフォルトのコンテキストウィンドウ（
num_ctx）は多くの場合わずか2,048トークンに設定されています。巨大なWikipediaページをローカルモデルに渡すと、プロンプトが静かに切り詰められ（指示が失われ）、空文字列が返される可能性があります！APIペイロードでOllamaに
"num_ctx": 32768を自動的に渡し、この切り詰めを防ぎます。Ollamaを使用する際は、ローカルマシンに32Kコンテキストウィンドウをサポートする十分なRAM/VRAMがあることを確認してください！
mcp.json）HTTP経由で実行する場合：
{
"mcpServers": {
"web-scraper": {
"url": "http://127.0.0.1:8000/mcp",
"headers": { "Authorization": "Bearer your_secure_token_here" }
}
}
}stdio（Docker）経由で実行する場合：
{
"mcpServers": {
"web-scraper": {
"command": "docker",
"args": [
"run", "-i", "--rm",
"-e", "SCRAPER_TRANSPORT=stdio",
"-e", "SCRAPER_OLLAMA_HOST=http://host.docker.internal:11434",
"-e", "SCRAPER_EXTRACT_MODEL=qwen3.5:2b",
"-e", "SCRAPER_RESEARCH_MODEL=qwen3.5:2b",
"PROG_UP_USERNAME/web-scraper-mcp:latest"
]
}
}
}すべての設定は環境変数（プレフィックス SCRAPER_）または .env ファイルで行います — .env.example を参照してください。
変数 | デフォルト | 備考 |
| （未設定） | HTTPエンドポイントのBearerトークン。ネットワーク展開では必須。未設定 = 認証なし（警告あり）。 |
|
| バインドアドレス。Dockerイメージはホストを |
|
| ヘッドレスページの同時実行上限（RAM/CPU依存）。 |
|
| クロールジョブのハード上限。 |
|
| ドメインごとのポライトなレート制限。 |
|
| robots.txtを尊重。 |
|
| falseのままにしてください — trueにするとSSRFガードが無効になります。 |
| （未設定） |
|
| （未設定） | オプションの検索バックエンド（最初に設定されたものが優先、それ以外はDuckDuckGo）。 |
SSRFガード — 取得するすべてのURL（および各リダイレクトホップ）はDNS解決され、プライベート/ループバック/リンクローカル/クラウドメタデータアドレスを指す場合は拒否されます。ブラウザはプライベートIPへのサブリソース要求も中止します。
認証 — HTTPトランスポートでのBearerトークン。デフォルトでlocalhostにバインド。
リソース上限 — レスポンスサイズ、タイムアウト、ページ同時実行数、クロールページ/深さの制限でホストを保護。
robots.txt + レート制限 がデフォルトで有効。
コンテナ — 非rootユーザーとして実行。シークレットは環境変数のみ。
CIはcosign（Sigstore）を使用してGitLabのOIDC IDでイメージにキーレス署名し、SPDX SBOMの証明書を添付します。実行前に検証してください：
cosign verify \
--certificate-oidc-issuer https://gitlab.cri.epita.fr \
--certificate-identity-regexp 'https://gitlab.cri.epita.fr/enzo.juhel/web-scraper//.*' \
registry.gitlab.cri.epita.fr/enzo.juhel/web-scraper@sha256:...benchmarks/run.py は、公開データセットとCrawl4AIベースラインに対して scrape/extract をスコアリングし、スコアカード（ページタイプごとのF1/精度 + 制限事項セクション）を出力します。ローカルまたは手動CIの benchmark ジョブで実行できます：
uv run python benchmarks/run.py --output scorecard.md有料プロキシ/CAPTCHAなし：高度に防御されたサイト（LinkedIn、Amazon、Cloudflareチャレンジ）はブロックされることがあります。ベンチマークのスコアカードでその箇所を定量化しています。
メインコンテンツ抽出は記事では強力ですが、フォーラム/製品/リストページでは弱いです（すべての抽出器に共通する既知の特性）。
インメモリのクロール/セッション状態 — 設計上、シングルプロセス、シングルユーザー。
uv run pre-commit install # local lint/secret hooks
uv run ruff check . && uv run ruff format --check .
uv run mypy src
uv run pytest -qSearch the web, read the top sources, and return a cited synthesis report.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | The research question. | |
| max_sources | No | How many top results to read. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the general workflow (searching the web, reading top sources, producing a cited report) but does not mention read-only behavior, potential latency, rate limits, or any limitations. The description is truthful and somewhat informative, but leaves important operational behavior unaddressed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence with a clear chronological structure, no filler, and immediate front-loading of the core action ('Search the web'). Every phrase earns its place: 'read the top sources' and 'cited synthesis report' are distinct and necessary components. This is appropriately concise for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the two fully documented parameters and the presence of an output schema (likely covering the report structure), the description covers the essential process and outcome well. What is missing is more explicit context about when to use the tool versus siblings and a few behavioral notes, but the combination of description and schema is largely sufficient for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%: both 'query' and 'max_sources' have descriptions in the schema. The tool description's phrasing ('research question', 'top results') mirrors the schema descriptions without adding new meaning about formats, constraints, or interpretation. This meets the baseline of 3 but does not exceed what structured data already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb sequence ('Search', 'read', 'return') with a clear deliverable ('a cited synthesis report'). It unambiguously identifies the tool's function and naturally distinguishes deep_research from siblings like search (just web results) or scrape/extract (page-level content gathering). The multi-step process is explicit and not a tautology.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for comprehensive research with cited synthesis, but it never explicitly states when to prefer this over search, scrape, or extract, nor does it give any 'instead of' guidance. Context makes the intended use reasonably clear, yet there are no exclusions or alternative routing as seen in higher-quality definitions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract structured data (JSON matching json_schema) or a text answer from a page.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to extract from. | |
| prompt | No | Natural-language extraction instruction. | |
| render | No | Force a browser render. | |
| json_schema | No | JSON Schema describing the fields to extract (recommended). |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry the behavioral burden. It does disclose the two operating modes (structured JSON versus text answer), which is meaningful. However, it does not mention rendering defaults, page-fetch behavior, rate limits, or side effects. The render parameter is defined in the schema, so some of that context is available, but the description itself stays minimal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with no wasted words. It could be slightly better structured by explicitly separating the two modes, but as written it is efficient and easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The output schema exists, so return-value details are covered elsewhere, and the input schema is fully documented. The main gaps are the lack of choice guidance between prompt and json_schema, and no indication of when to use extract instead of scrape. These gaps make the description adequate but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already defines url, prompt, render, and json_schema. The description adds a high-level mapping by linking json_schema to structured JSON and implying the text answer comes from the prompt, but it does not add much beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('extract'), the target resource ('a page'), and the two possible outputs: JSON matching json_schema or a text answer. It does not explicitly distinguish itself from sibling tools like scrape or search, though the verb and output specification make the core purpose clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no guidance on when to use extract versus scrape, search, or deep_research, and no exclusions or conditions. The only context is 'from a page,' which is too thin to help an agent choose between this and its siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scrape a single URL into clean markdown (boilerplate/ads stripped).
Tries a fast static fetch first and falls back to a stealth headless browser automatically when the page looks JS-gated.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to scrape (http/https). | |
| render | No | Force a headless browser render (for JS-heavy pages). | |
| include_links | No | Also return all links found on the page. | |
| include_raw_html | No | Also return the raw HTML (large). |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of behavioral disclosure. It clearly reveals the two-phase fetch strategy, the automatic fallback to headless browsing, and the output transformation to clean markdown. It does not discuss rate limits, authentication, or failure behavior, but the core operational traits are transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with no fluff, front-loading the primary purpose before the fallback behavior. Every sentence contributes useful information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The presence of an output schema means return values do not need to be explained in the description. Combined with fully documented parameters and a clear explanation of the scraping strategy, the definition is largely complete, though it omits failure semantics and any rate-limit or authentication caveats.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all four parameters are already documented in the input schema. The tool description adds no additional parameter-level meaning, which matches the baseline for fully-covered schemas.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Scrape a single URL') and a concrete outcome ('into clean markdown'), making the tool's purpose immediately clear. The boilerplate/ads-stripping detail further distinguishes it from general fetch or research siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explains that a fast static fetch is attempted first and that a stealth headless browser is used automatically for JS-gated pages, giving clear context on when the tool adapts. It does not explicitly name alternatives or say when to prefer sibling tools, so it stops short of full exclusion guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Web search. Returns ranked results (title, url, content snippet).
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | Search query. | |
| max_results | No | Max results. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description carries the full burden. It does communicate the read-only nature implicitly by saying 'Returns', but it doesn't disclose rate limits, result freshness, error behavior, or whether any resources are modified. This is additional but minimal context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One short sentence front-loads the primary action and return format. No filler words; the description is appropriately sized and every word contributes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool with a complete input schema and an output schema, the description is largely sufficient. It identifies what the tool returns and the parameters are well covered by the schema. It lacks any mention of when to use it relative to deep_research, but that falls under usage guidance rather than completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with descriptions for both query and max_results. The tool description adds no parameter-level information beyond the schema, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-resource pair ('Web search') and specifies the return shape (ranked results with title, url, content snippet). This clearly distinguishes it from sibling tools like scrape, extract, and deep_research without needing to open their schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to choose search over its siblings, nor any exclusions or alternative recommendations. The presence of deep_research, scrape, and extract in the same toolset makes this omission a real gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0deep_researchextractscrapesearchScored across 4 tools
scrape and extract both operate on a page but are clearly distinguished by output type: markdown document vs structured JSON/answer. search and deep_research are also distinct, though deep_research could be seen as a superset of search.
All tools use lowercase snake_case imperative verbs: scrape, extract, search, deep_research. The naming pattern is consistent and predictable.
Four tools is well-scoped for a web scraper/research server: one for raw page content, one for structured extraction, one for search, and one for synthesis. Each tool has a clear purpose with no redundant bloat.
The core web research workflow is covered: search, fetch, extract, and synthesize. A minor gap is the lack of batch crawling or link-following features, but most common agent tasks can be completed with these tools.
Live web access for agents: scrape, SERP search, crawl/map, 100+ collectors, datasets, proxies.
Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili…
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
