셀프 호스팅 Model Context Protocol 서버로, LLM 클라이언트(Claude Code, Cursor, ChatGPT)에게 유료 스크래핑 서비스와 동일한 도구 표면을 제공합니다 — scrape, crawl, map, search, extract, interact, deep_research — 전적으로 자체 하드웨어에서 실행됩니다.
유료 프록시/CAPTCHA 서비스 없음: 안티봇은 자체 호스팅됩니다(헤드리스 Chromium + playwright-stealth, robots.txt, 예의 있는 속도 제한). 강력하게 보호된 사이트는 여전히 차단될 수 있습니다. 제한 사항을 참조하세요.
도구 | 기능 |
| 하나의 URL → 깨끗한 마크다운(보일러플레이트 제거). 정적 우선, JS 페이지용 자동 브라우저 폴백. |
| 백그라운드 BFS 크롤 작업(중복 제거, 깊이/페이지 상한); 결과를 폴링. |
| 페이지의 링크 목록(선택적으로 동일 도메인) — 무엇을 크롤할지 결정. |
| 웹 검색. 플러그형 백엔드: DuckDuckGo(기본), SearXNG, Brave, 또는 Tavily. |
| 페이지를 가져와서 스키마에 맞는 구조화된 JSON을 LLM을 통해 추출. |
| 토큰 효율적인 ARIA 스냅샷을 사용하여 영구 브라우저 세션(클릭/입력/키 누름)을 구동. |
| 검색 → 상위 소스 읽기 → 인용된 종합 보고서 반환. |
extract 및 deep_research에는 ANTHROPIC_API_KEY가 필요합니다.
Related MCP server: Fetch MCP Server
uv sync # install deps (uses the pinned uv.lock)
uv run playwright install chromium
export SCRAPER_AUTH_TOKEN=$(openssl rand -hex 32)
uv run web-scraper-mcp # HTTP server on http://127.0.0.1:8000/mcpStdio(로컬, 데스크톱 클라이언트용): SCRAPER_TRANSPORT=stdio uv run web-scraper-mcp.
시작하는 가장 빠른 방법은 DockerHub에서 사전 빌드된 이미지를 가져오는 것입니다.
# Pull the latest image
docker pull PROG_UP_USERNAME/web-scraper-mcp:latest
# Run the container (with Anthropic / Claude)
docker run -p 8000:8000 \
-e SCRAPER_AUTH_TOKEN=your_secure_token_here \
-e ANTHROPIC_API_KEY=sk-ant-api03-... \
PROG_UP_USERNAME/web-scraper-mcp:latest
# Or run the container over stdio (useful for local MCP clients)
docker run -i --rm \
-e SCRAPER_TRANSPORT=stdio \
-e SCRAPER_AUTH_TOKEN=your_secure_token_here \
PROG_UP_USERNAME/web-scraper-mcp:latest(실제 DockerHub 사용자 이름으로 PROG_UP_USERNAME을 교체해야 합니다).
Anthropic API 대신 로컬에서 모델을 실행하려는 경우, 서버는 extract 및 deep_research 도구를 위한 대체 백엔드로 Ollama를 완전히 지원합니다.
docker run -p 8000:8000 \
-e SCRAPER_AUTH_TOKEN=your_secure_token_here \
-e SCRAPER_OLLAMA_HOST=http://host.docker.internal:11434 \
-e SCRAPER_EXTRACT_MODEL=qwen3.5:2b \
-e SCRAPER_RESEARCH_MODEL=qwen3.5:2b \
PROG_UP_USERNAME/web-scraper-mcp:latest[!WARNING] 컨텍스트 창은 중요합니다! 웹 스크래핑은 엄청난 양의 마크다운을 생성합니다.
extract는 최대 100,000자,deep_research는 최대 16,000자를 LLM에 보낼 수 있습니다.기본적으로 Claude는 대규모 컨텍스트를 기본적으로 처리합니다. 그러나 Ollama의 기본 컨텍스트 창(
num_ctx)은 종종 2,048 토큰으로 구성됩니다. 거대한 Wikipedia 페이지를 로컬 모델에 전달하면 프롬프트가 조용히 잘려나가(지침이 누락됨) 빈 문자열이 반환됩니다!우리는 API 페이로드에
"num_ctx": 32768을 자동으로 전달하여 이러한 잘림을 방지합니다. Ollama를 사용할 때 로컬 머신에 32K 컨텍스트 창을 지원할 충분한 RAM/VRAM이 있는지 확인하세요!
mcp.json)HTTP로 실행하는 경우:
{
"mcpServers": {
"web-scraper": {
"url": "http://127.0.0.1:8000/mcp",
"headers": { "Authorization": "Bearer your_secure_token_here" }
}
}
}stdio(Docker)로 실행하는 경우:
{
"mcpServers": {
"web-scraper": {
"command": "docker",
"args": [
"run", "-i", "--rm",
"-e", "SCRAPER_TRANSPORT=stdio",
"-e", "SCRAPER_OLLAMA_HOST=http://host.docker.internal:11434",
"-e", "SCRAPER_EXTRACT_MODEL=qwen3.5:2b",
"-e", "SCRAPER_RESEARCH_MODEL=qwen3.5:2b",
"PROG_UP_USERNAME/web-scraper-mcp:latest"
]
}
}
}모든 설정은 환경 변수(접두사 SCRAPER_) 또는 .env 파일입니다 — .env.example 참조.
변수 | 기본값 | 참고 |
| (설정 안 됨) | HTTP 엔드포인트용 Bearer 토큰. 네트워크 배포에는 필수; 설정 안 됨 = 인증 없음(경고). |
|
| 바인드 주소. Docker 이미지는 호스트 |
|
| 헤드리스 페이지 동시성 상한(RAM/CPU 제한). |
|
| 크롤 작업의 하드 상한. |
|
| 도메인별 예의 있는 속도 제한. |
|
| robots.txt 준수. |
|
| false 유지 — true이면 SSRF 가드를 비활성화합니다. |
| (설정 안 됨) |
|
| (설정 안 됨) | 선택적 검색 백엔드(첫 번째 설정이 우선, 그 외에는 DuckDuckGo). |
SSRF 가드 — 가져온 모든 URL(및 각 리디렉션 홉)은 DNS로 확인되며, 개인/루프백/링크-로컬/클라우드 메타데이터 주소를 가리키면 거부됩니다. 브라우저는 또한 개인 IP에 대한 하위 리소스 요청을 중단합니다.
인증 — HTTP 전송에 Bearer 토큰; 기본적으로 localhost에 바인딩.
리소스 상한 — 응답 크기, 시간 초과, 페이지 동시성, 크롤 페이지/깊이 제한으로 호스트를 보호합니다.
robots.txt + 속도 제한 기본적으로 켜져 있습니다.
컨테이너 — 비루트 사용자로 실행; 비밀은 환경 변수로만 전달.
CI는 GitLab의 OIDC ID를 사용하여 cosign(Sigstore)으로 이미지에 키 없는 서명을 하고, SPDX SBOM 증명을 첨부합니다. 실행 전에 검증하세요:
cosign verify \
--certificate-oidc-issuer https://gitlab.cri.epita.fr \
--certificate-identity-regexp 'https://gitlab.cri.epita.fr/enzo.juhel/web-scraper//.*' \
registry.gitlab.cri.epita.fr/enzo.juhel/web-scraper@sha256:...benchmarks/run.py는 공개 데이터 세트와 Crawl4AI 기준선에 대해 scrape/extract를 평가하여 스코어카드(페이지 유형별 F1/정확도 + 제한 사항 섹션)를 생성합니다. 로컬 또는 수동 CI benchmark 작업으로 실행:
uv run python benchmarks/run.py --output scorecard.md유료 프록시/CAPTCHA 없음: 강력하게 보호된 사이트(LinkedIn, Amazon, Cloudflare 챌린지)는 때때로 차단됩니다. 벤치마크 스코어카드가 어디서 그런지 정량화합니다.
주요 콘텐츠 추출은 기사에서 강하고, 포럼/제품/목록 페이지에서는 약합니다(모든 추출기의 알려진 속성).
메모리 내 크롤/세션 상태 — 설계상 단일 프로세스, 단일 사용자.
uv run pre-commit install # local lint/secret hooks
uv run ruff check . && uv run ruff format --check .
uv run mypy src
uv run pytest -qSearch the web, read the top sources, and return a cited synthesis report.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | The research question. | |
| max_sources | No | How many top results to read. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the general workflow (searching the web, reading top sources, producing a cited report) but does not mention read-only behavior, potential latency, rate limits, or any limitations. The description is truthful and somewhat informative, but leaves important operational behavior unaddressed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence with a clear chronological structure, no filler, and immediate front-loading of the core action ('Search the web'). Every phrase earns its place: 'read the top sources' and 'cited synthesis report' are distinct and necessary components. This is appropriately concise for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the two fully documented parameters and the presence of an output schema (likely covering the report structure), the description covers the essential process and outcome well. What is missing is more explicit context about when to use the tool versus siblings and a few behavioral notes, but the combination of description and schema is largely sufficient for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%: both 'query' and 'max_sources' have descriptions in the schema. The tool description's phrasing ('research question', 'top results') mirrors the schema descriptions without adding new meaning about formats, constraints, or interpretation. This meets the baseline of 3 but does not exceed what structured data already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb sequence ('Search', 'read', 'return') with a clear deliverable ('a cited synthesis report'). It unambiguously identifies the tool's function and naturally distinguishes deep_research from siblings like search (just web results) or scrape/extract (page-level content gathering). The multi-step process is explicit and not a tautology.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for comprehensive research with cited synthesis, but it never explicitly states when to prefer this over search, scrape, or extract, nor does it give any 'instead of' guidance. Context makes the intended use reasonably clear, yet there are no exclusions or alternative routing as seen in higher-quality definitions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract structured data (JSON matching json_schema) or a text answer from a page.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to extract from. | |
| prompt | No | Natural-language extraction instruction. | |
| render | No | Force a browser render. | |
| json_schema | No | JSON Schema describing the fields to extract (recommended). |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry the behavioral burden. It does disclose the two operating modes (structured JSON versus text answer), which is meaningful. However, it does not mention rendering defaults, page-fetch behavior, rate limits, or side effects. The render parameter is defined in the schema, so some of that context is available, but the description itself stays minimal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with no wasted words. It could be slightly better structured by explicitly separating the two modes, but as written it is efficient and easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The output schema exists, so return-value details are covered elsewhere, and the input schema is fully documented. The main gaps are the lack of choice guidance between prompt and json_schema, and no indication of when to use extract instead of scrape. These gaps make the description adequate but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already defines url, prompt, render, and json_schema. The description adds a high-level mapping by linking json_schema to structured JSON and implying the text answer comes from the prompt, but it does not add much beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('extract'), the target resource ('a page'), and the two possible outputs: JSON matching json_schema or a text answer. It does not explicitly distinguish itself from sibling tools like scrape or search, though the verb and output specification make the core purpose clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no guidance on when to use extract versus scrape, search, or deep_research, and no exclusions or conditions. The only context is 'from a page,' which is too thin to help an agent choose between this and its siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scrape a single URL into clean markdown (boilerplate/ads stripped).
Tries a fast static fetch first and falls back to a stealth headless browser automatically when the page looks JS-gated.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to scrape (http/https). | |
| render | No | Force a headless browser render (for JS-heavy pages). | |
| include_links | No | Also return all links found on the page. | |
| include_raw_html | No | Also return the raw HTML (large). |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of behavioral disclosure. It clearly reveals the two-phase fetch strategy, the automatic fallback to headless browsing, and the output transformation to clean markdown. It does not discuss rate limits, authentication, or failure behavior, but the core operational traits are transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with no fluff, front-loading the primary purpose before the fallback behavior. Every sentence contributes useful information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The presence of an output schema means return values do not need to be explained in the description. Combined with fully documented parameters and a clear explanation of the scraping strategy, the definition is largely complete, though it omits failure semantics and any rate-limit or authentication caveats.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all four parameters are already documented in the input schema. The tool description adds no additional parameter-level meaning, which matches the baseline for fully-covered schemas.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Scrape a single URL') and a concrete outcome ('into clean markdown'), making the tool's purpose immediately clear. The boilerplate/ads-stripping detail further distinguishes it from general fetch or research siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explains that a fast static fetch is attempted first and that a stealth headless browser is used automatically for JS-gated pages, giving clear context on when the tool adapts. It does not explicitly name alternatives or say when to prefer sibling tools, so it stops short of full exclusion guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Web search. Returns ranked results (title, url, content snippet).
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | Search query. | |
| max_results | No | Max results. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description carries the full burden. It does communicate the read-only nature implicitly by saying 'Returns', but it doesn't disclose rate limits, result freshness, error behavior, or whether any resources are modified. This is additional but minimal context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One short sentence front-loads the primary action and return format. No filler words; the description is appropriately sized and every word contributes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool with a complete input schema and an output schema, the description is largely sufficient. It identifies what the tool returns and the parameters are well covered by the schema. It lacks any mention of when to use it relative to deep_research, but that falls under usage guidance rather than completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with descriptions for both query and max_results. The tool description adds no parameter-level information beyond the schema, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-resource pair ('Web search') and specifies the return shape (ranked results with title, url, content snippet). This clearly distinguishes it from sibling tools like scrape, extract, and deep_research without needing to open their schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to choose search over its siblings, nor any exclusions or alternative recommendations. The presence of deep_research, scrape, and extract in the same toolset makes this omission a real gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0deep_researchextractscrapesearchScored across 4 tools
scrape and extract both operate on a page but are clearly distinguished by output type: markdown document vs structured JSON/answer. search and deep_research are also distinct, though deep_research could be seen as a superset of search.
All tools use lowercase snake_case imperative verbs: scrape, extract, search, deep_research. The naming pattern is consistent and predictable.
Four tools is well-scoped for a web scraper/research server: one for raw page content, one for structured extraction, one for search, and one for synthesis. Each tool has a clear purpose with no redundant bloat.
The core web research workflow is covered: search, fetch, extract, and synthesize. A minor gap is the lack of batch crawling or link-following features, but most common agent tasks can be completed with these tools.
Live web access for agents: scrape, SERP search, crawl/map, 100+ collectors, datasets, proxies.
Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili…
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
