⚠️ 지원 중단됨 —
@robot-resources/scraper에 통합되었습니다. 이 MCP 서버는 이제 메인 스크레이퍼 패키지 내부에scraper-mcp바이너리로 포함되어 제공됩니다. 패키지 하나만 설치하면 MCP를 사용할 수 있습니다.npx robot-resources --for=cursor또는npx robot-resources --for=claude-code를 통해 자동 설정이 가능합니다. 아래의 도구(scraper_compress_url,scraper_crawl_url)와 매개변수는 변경되지 않았습니다.
Scraper용 MCP 서버 — AI 에이전트를 위한 컨텍스트 압축.
AI 에이전트를 위한 인사 관리(Human Resources)입니다.
Robot Resources는 AI 에이전트에게 두 가지 강력한 기능을 제공합니다:
라우터(Router) — 각 LLM 호출을 가장 저렴하고 유능한 모델로 라우팅합니다. OpenAI, Anthropic, Google 전반에서 60-90%의 비용 절감 효과를 제공합니다.
스크레이퍼(Scraper) — 웹 페이지를 깔끔한 마크다운으로 압축합니다. 페이지당 토큰을 70-80% 절감합니다.
모두 로컬에서 실행됩니다. API 키는 절대 기기 밖으로 나가지 않습니다. 무료, 무제한, 티어 제한 없음.
npx robot-resources명령어 하나로 모든 설정이 완료됩니다. 자세한 내용은 robotresources.ai에서 확인하세요.
Related MCP server: Scraper MCP
이 패키지는 Model Context Protocol을 통해 AI 에이전트가 웹 콘텐츠를 토큰 효율적인 마크다운으로 압축할 수 있는 두 가지 도구(단일 페이지 압축 및 다중 페이지 BFS 크롤링)를 제공합니다.
npx @robot-resources/scraper-mcp또는 전역으로 설치:
npm install -g @robot-resources/scraper-mcpclaude_desktop_config.json에 추가하세요:
{
"mcpServers": {
"scraper": {
"command": "npx",
"args": ["-y", "@robot-resources/scraper-mcp"]
}
}
}scraper_compress_url단일 웹 페이지를 70-90% 더 적은 토큰의 마크다운으로 압축합니다.
매개변수:
매개변수 | 타입 | 필수 | 기본값 | 설명 |
| string | 예 | — | 압축할 URL |
| string | 아니오 |
|
|
| number | 아니오 |
| 페치 타임아웃 (밀리초) |
| number | 아니오 |
| 최대 재시도 횟수 (0-10) |
예시 프롬프트: "Compress https://docs.example.com/getting-started"
scraper_crawl_urlBFS 링크 탐색을 사용하여 시작 URL에서 여러 페이지를 크롤링합니다.
매개변수:
매개변수 | 타입 | 필수 | 기본값 | 설명 |
| string | 예 | — | 크롤링을 시작할 URL |
| number | 아니오 |
| 최대 크롤링 페이지 수 (1-100) |
| number | 아니오 |
| 최대 링크 깊이 (0-5) |
| string | 아니오 |
|
|
| string[] | 아니오 | — | 포함할 URL 패턴 (glob) |
| string[] | 아니오 | — | 제외할 URL 패턴 (glob) |
| number | 아니오 |
| 페이지당 타임아웃 (밀리초) |
예시 프롬프트: "Crawl the docs at https://docs.example.com with max 20 pages"
모드 | 방식 | 사용 시기 |
| 일반 HTTP | 기본 사이트, API, 문서 |
| TLS 지문 위장 | 봇 방지 사이트 |
| 헤드리스 브라우저 (Playwright) | JS 렌더링 SPA |
| Fast → 403/챌린지 발생 시 stealth로 폴백 | 알 수 없는 사이트 (기본값) |
Stealth 모드는 impit이 필요하며, Render 모드는 @robot-resources/scraper의 피어 의존성으로 playwright가 필요합니다.
Node.js 18+
@robot-resources/scraper - 핵심 압축 라이브러리
@robot-resources/router-mcp - LLM 비용 최적화를 위한 MCP 서버
Robot Resources - AI 에이전트를 위한 인사 관리
MIT
Compress web content from a URL for reduced token usage. Returns markdown with 70-90% fewer tokens than raw HTML.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to compress | |
| mode | No | Fetch mode: 'fast' (plain HTTP), 'stealth' (TLS fingerprint), 'render' (headless browser), 'auto' (fast with fallback). Default: 'auto' | |
| timeout | No | Fetch timeout in milliseconds (default: 10000) | |
| maxRetries | No | Max retry attempts (default: 3) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the token reduction benefit but fails to disclose critical traits like rate limits, authentication needs, error handling, or what happens with invalid URLs. The description is minimal and does not compensate for the lack of annotations, leaving significant gaps in understanding the tool's behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is highly concise and front-loaded, consisting of two sentences that directly state the tool's function and outcome without any fluff. Every sentence earns its place by delivering essential information efficiently, making it easy to grasp the tool's purpose quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of a tool with 4 parameters, no annotations, and no output schema, the description is incomplete. It lacks details on behavioral aspects, error cases, and output format beyond 'markdown,' failing to provide sufficient context for safe and effective use. The description does not compensate for the absence of structured data, leaving key operational questions unanswered.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% description coverage, providing clear details for all parameters (url, mode, timeout, maxRetries). The description adds no additional parameter semantics beyond what the schema already explains, such as usage examples or implications of mode choices. Thus, it meets the baseline score of 3, as the schema adequately covers parameter information.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose with a specific verb ('compress') and resource ('web content from a URL'), and distinguishes it from its sibling 'scraper_crawl_url' by focusing on token reduction rather than crawling. It explicitly mentions the outcome: 'Returns markdown with 70-90% fewer tokens than raw HTML.'
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus its sibling 'scraper_crawl_url' or any alternatives. It lacks context about prerequisites, such as when compression is beneficial or when to choose specific modes, offering only a basic functional statement without comparative or exclusionary advice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Crawl multiple pages from a starting URL using BFS link discovery. Returns compressed markdown for each page with 70-90% fewer tokens than raw HTML.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL to crawl | |
| maxPages | No | Max pages to crawl (default: 10) | |
| maxDepth | No | Max link depth (default: 2) | |
| mode | No | Fetch mode: 'fast' (plain HTTP), 'stealth' (TLS fingerprint), 'render' (headless browser), 'auto' (fast with fallback). Default: 'auto' | |
| include | No | URL patterns to include (glob) | |
| exclude | No | URL patterns to exclude (glob) | |
| timeout | No | Per-page timeout in milliseconds (default: 10000) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: the BFS crawling algorithm, multi-page scope, and the 70-90% token reduction in output. It doesn't mention rate limits, authentication needs, or error handling, but provides substantial operational context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences that efficiently convey the core functionality and key output characteristic. Every word earns its place with no redundancy or unnecessary elaboration. The description is front-loaded with the primary action and scope.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter tool with no annotations and no output schema, the description provides good operational context but lacks details about return format structure, error conditions, or performance characteristics. It mentions compressed markdown output but doesn't describe the data structure or pagination approach.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds no parameter-specific information beyond what's already in the schema descriptions. It mentions 'starting URL' which is covered by the url parameter description, but doesn't elaborate on parameter interactions or usage patterns.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose with specific verbs ('crawl multiple pages', 'returns compressed markdown') and resources ('starting URL', 'BFS link discovery'). It distinguishes from the sibling tool scraper_compress_url by emphasizing multi-page crawling rather than single-page compression.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context through 'crawl multiple pages' and 'starting URL', suggesting this is for web scraping tasks. However, it doesn't explicitly state when to use this tool versus the sibling scraper_compress_url or other alternatives, nor does it provide exclusion criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.2scraper_compress_urlscraper_crawl_urlScored across 2 tools
The two tools have clearly distinct purposes: scraper_compress_url handles single-page compression, while scraper_crawl_url performs multi-page crawling with BFS link discovery. There is no overlap or ambiguity between them.
Both tools follow a consistent verb_noun pattern (scraper_compress_url and scraper_crawl_url), using the same prefix and snake_case style throughout. The naming is predictable and uniform.
With only two tools, the server feels thin for a scraper domain, lacking essential operations like error handling, configuration, or content filtering. This minimal set may limit agent workflows and cause dead ends.
The tool surface is severely incomplete for web scraping; it misses basic CRUD-like operations such as updating crawl parameters, deleting cached data, or retrieving specific page elements. Agents will struggle with incomplete coverage.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Web pages and whole sites to clean Markdown for RAG and AI agents. Crawls links and sitemaps.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Web scraping for AI agents: scrape, search, crawl, map any website to markdown + JSON. No browser.
