# Clone the repository
git clone https://github.com/webscraping-ai/webscraping-ai-mcp-server.git
cd webscraping-ai-mcp-server

# Install dependencies
npm install

# Run
npm start

커서에서 구성

참고: Cursor 버전 0.45.6+가 필요합니다.

WebScraping.AI MCP 서버는 Cursor에서 두 가지 방법으로 구성할 수 있습니다.

프로젝트별 구성 (팀 프로젝트에 권장): 프로젝트 디렉토리에 .cursor/mcp.json 파일을 만듭니다.

{
  "servers": {
    "webscraping-ai": {
      "type": "command",
      "command": "npx -y webscraping-ai-mcp",
      "env": {
        "WEBSCRAPING_AI_API_KEY": "your-api-key",
        "WEBSCRAPING_AI_CONCURRENCY_LIMIT": "5"
      }
    }
  }
}

전역 구성 (모든 프로젝트에서 개인적으로 사용): 위와 동일한 구성 형식으로 홈 디렉토리에 ~/.cursor/mcp.json 파일을 만듭니다.

Windows를 사용 중이고 문제가 발생하는 경우 명령으로 cmd /c "set WEBSCRAPING_AI_API_KEY=your-api-key && npx -y webscraping-ai-mcp" 사용해 보세요.

이 구성을 사용하면 웹 스크래핑 작업과 관련된 경우 WebScraping.AI 도구가 Cursor의 AI 에이전트에서 자동으로 사용 가능하게 됩니다.

Claude Desktop에서 실행

claude_desktop_config.json 에 다음을 추가하세요:

{
  "mcpServers": {
    "mcp-server-webscraping-ai": {
      "command": "npx",
      "args": ["-y", "webscraping-ai-mcp"],
      "env": {
        "WEBSCRAPING_AI_API_KEY": "YOUR_API_KEY_HERE",
        "WEBSCRAPING_AI_CONCURRENCY_LIMIT": "5"
      }
    }
  }
}

구성

환경 변수

필수의

WEBSCRAPING_AI_API_KEY : WebScraping.AI API 키
- 모든 작업에 필요합니다
- WebScraping.AI 에서 API 키를 받으세요

선택적 구성

WEBSCRAPING_AI_CONCURRENCY_LIMIT : 동시 요청 최대 수(기본값: 5 )
WEBSCRAPING_AI_DEFAULT_PROXY_TYPE : 사용할 프록시 유형(기본값: residential )
WEBSCRAPING_AI_DEFAULT_JS_RENDERING : JavaScript 렌더링을 활성화/비활성화합니다(기본값: true )
WEBSCRAPING_AI_DEFAULT_TIMEOUT : 웹 페이지 검색 최대 시간(ms) (기본값: 15000 , 최대값: 30000 )
WEBSCRAPING_AI_DEFAULT_JS_TIMEOUT : 최대 JavaScript 렌더링 시간(ms) (기본값: 2000 )

구성 예제

표준 사용의 경우:

# Required
export WEBSCRAPING_AI_API_KEY=your-api-key

# Optional - customize behavior (default values)
export WEBSCRAPING_AI_CONCURRENCY_LIMIT=5
export WEBSCRAPING_AI_DEFAULT_PROXY_TYPE=residential # datacenter or residential
export WEBSCRAPING_AI_DEFAULT_JS_RENDERING=true
export WEBSCRAPING_AI_DEFAULT_TIMEOUT=15000
export WEBSCRAPING_AI_DEFAULT_JS_TIMEOUT=2000

사용 가능한 도구

1. 질문 도구( `webscraping_ai_question` )

웹 페이지 콘텐츠에 대한 질문을 해보세요.

{
  "name": "webscraping_ai_question",
  "arguments": {
    "url": "https://example.com",
    "question": "What is the main topic of this page?",
    "timeout": 30000,
    "js": true,
    "js_timeout": 2000,
    "wait_for": ".content-loaded",
    "proxy": "datacenter",
    "country": "us"
  }
}

응답 예시:

{
  "content": [
    {
      "type": "text",
      "text": "The main topic of this page is examples and documentation for HTML and web standards."
    }
  ],
  "isError": false
}

2. 필드 도구( `webscraping_ai_fields` )

지침에 따라 웹 페이지에서 구조화된 데이터를 추출합니다.

{
  "name": "webscraping_ai_fields",
  "arguments": {
    "url": "https://example.com/product",
    "fields": {
      "title": "Extract the product title",
      "price": "Extract the product price",
      "description": "Extract the product description"
    },
    "js": true,
    "timeout": 30000
  }
}

응답 예시:

{
  "content": [
    {
      "type": "text",
      "text": {
        "title": "Example Product",
        "price": "$99.99",
        "description": "This is an example product description."
      }
    }
  ],
  "isError": false
}

3. HTML 도구( `webscraping_ai_html` )

JavaScript 렌더링을 통해 웹 페이지의 전체 HTML을 얻습니다.

{
  "name": "webscraping_ai_html",
  "arguments": {
    "url": "https://example.com",
    "js": true,
    "timeout": 30000,
    "wait_for": "#content-loaded"
  }
}

응답 예시:

{
  "content": [
    {
      "type": "text",
      "text": "<html>...[full HTML content]...</html>"
    }
  ],
  "isError": false
}

4. 텍스트 도구( `webscraping_ai_text` )

웹 페이지에서 보이는 텍스트 콘텐츠를 추출합니다.

{
  "name": "webscraping_ai_text",
  "arguments": {
    "url": "https://example.com",
    "js": true,
    "timeout": 30000
  }
}

응답 예시:

{
  "content": [
    {
      "type": "text",
      "text": "Example Domain\nThis domain is for use in illustrative examples in documents..."
    }
  ],
  "isError": false
}

5. 선택된 도구( `webscraping_ai_selected` )

CSS 선택기를 사용하여 특정 요소에서 콘텐츠를 추출합니다.

{
  "name": "webscraping_ai_selected",
  "arguments": {
    "url": "https://example.com",
    "selector": "div.main-content",
    "js": true,
    "timeout": 30000
  }
}

응답 예시:

{
  "content": [
    {
      "type": "text",
      "text": "<div class=\"main-content\">This is the main content of the page.</div>"
    }
  ],
  "isError": false
}

6. 선택된 다중 도구( `webscraping_ai_selected_multiple` )

CSS 선택기를 사용하여 여러 요소에서 콘텐츠를 추출합니다.

{
  "name": "webscraping_ai_selected_multiple",
  "arguments": {
    "url": "https://example.com",
    "selectors": ["div.header", "div.product-list", "div.footer"],
    "js": true,
    "timeout": 30000
  }
}

응답 예시:

{
  "content": [
    {
      "type": "text",
      "text": [
        "<div class=\"header\">Header content</div>",
        "<div class=\"product-list\">Product list content</div>",
        "<div class=\"footer\">Footer content</div>"
      ]
    }
  ],
  "isError": false
}

7. 계정 도구( `webscraping_ai_account` )

WebScraping.AI 계정에 대한 정보를 얻으세요.

{
  "name": "webscraping_ai_account",
  "arguments": {}
}

응답 예시:

{
  "content": [
    {
      "type": "text",
      "text": {
        "requests": 5000,
        "remaining": 4500,
        "limit": 10000,
        "resets_at": "2023-12-31T23:59:59Z"
      }
    }
  ],
  "isError": false
}

모든 도구에 대한 공통 옵션

다음 옵션은 모든 스크래핑 도구에서 사용할 수 있습니다.

timeout : 웹 페이지 검색에 걸리는 최대 시간(ms) (기본값 15000, 최대값 30000)
js : 헤드리스 브라우저를 사용하여 온페이지 JavaScript를 실행합니다(기본값은 true)
js_timeout : 최대 JavaScript 렌더링 시간(ms) (기본값 2000)
wait_for : 페이지 콘텐츠를 반환하기 전에 기다릴 CSS 선택기
proxy : 프록시 유형, 데이터 센터 또는 주거용(기본값은 주거용)
country : 사용할 프록시 국가(기본값: 미국). 지원 국가: us, gb, de, it, fr, ca, es, ru, jp, kr, in
custom_proxy : " http://user:password@host:port " 형식의 사용자 고유 프록시 URL
device : 장치 에뮬레이션 유형입니다. 지원되는 값: 데스크톱, 모바일, 태블릿
error_on_404 : 대상 페이지에서 404 HTTP 상태에 대한 오류를 반환합니다(기본값은 false)
error_on_redirect : 대상 페이지에서 리디렉션 시 오류를 반환합니다(기본값은 false)
js_script : 대상 페이지에서 실행할 사용자 정의 JavaScript 코드

오류 처리

서버는 강력한 오류 처리 기능을 제공합니다.

일시적인 오류에 대한 자동 재시도
백오프를 통한 속도 제한 처리
자세한 오류 메시지
네트워크 복원력

오류 응답 예:

{
  "content": [
    {
      "type": "text",
      "text": "API Error: 429 Too Many Requests"
    }
  ],
  "isError": true
}

LLM과의 통합

이 서버는 모델 컨텍스트 프로토콜(Model Context Protocol)을 구현하여 모든 MCP 지원 LLM 플랫폼과 호환됩니다. 웹 스크래핑 작업에 이러한 도구를 사용하도록 LLM을 구성할 수 있습니다.

예: MCP를 사용하여 Claude 구성

const { Claude } = require('@anthropic-ai/sdk');
const { Client } = require('@modelcontextprotocol/sdk/client/index.js');
const { StdioClientTransport } = require('@modelcontextprotocol/sdk/client/stdio.js');

const claude = new Claude({
  apiKey: process.env.ANTHROPIC_API_KEY
});

const transport = new StdioClientTransport({
  command: 'npx',
  args: ['-y', 'webscraping-ai-mcp'],
  env: {
    WEBSCRAPING_AI_API_KEY: 'your-api-key'
  }
});

const client = new Client({
  name: 'claude-client',
  version: '1.0.0'
});

await client.connect(transport);

// Now you can use Claude with WebScraping.AI tools
const tools = await client.listTools();
const response = await claude.complete({
  prompt: 'What is the main topic of example.com?',
  tools: tools
});

개발

# Clone the repository
git clone https://github.com/webscraping-ai/webscraping-ai-mcp-server.git
cd webscraping-ai-mcp-server

# Install dependencies
npm install

# Run tests
npm test

# Add your .env file
cp .env.example .env

# Start the inspector
npx @modelcontextprotocol/inspector node src/index.js

기여하다

저장소를 포크하세요
기능 브랜치를 생성하세요
테스트 실행: npm test
풀 리퀘스트 제출

특허

MIT 라이센스 - 자세한 내용은 라이센스 파일을 참조하세요.

Available Tools

7 tools

webscraping_ai_accountD

ParametersJSON Schema

Name	Required	Description	Default
No parameters

TDQS

D1/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Tool has no description.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness1/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Tool has no description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Tool has no description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Tool has no description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose1/5

Does the description clearly state what the tool does and how it differs from similar tools?

Tool has no description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Tool has no description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

webscraping_ai_fieldsD

ParametersJSON Schema

Name	Required	Description	Default
`js`	No	Execute on-page JavaScript using a headless browser (false by default).
`url`	Yes	URL of the target page.
`proxy`	No	Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page.	datacenter
`device`	No	Type of device emulation.
`fields`	Yes	Dictionary of field names with instructions for extraction.
`country`	No	Country of the proxy to use (US by default).
`timeout`	No	Maximum web page retrieval time in ms (20000 by default, maximum is 30000).
`wait_for`	No	CSS selector to wait for before returning the page content.
`js_script`	No	Custom JavaScript code to execute on the target page.
`js_timeout`	No	Maximum JavaScript rendering time in ms (3000 by default).
`custom_proxy`	No	Your own proxy URL in "http://user:password@host:port" format.
`error_on_404`	No	Return error on 404 HTTP status on the target page (false by default).
`error_on_redirect`	No	Return error on redirect on the target page (false by default).

TDQS

D1/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Tool has no description.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness1/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Tool has no description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Tool has no description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Tool has no description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose1/5

Does the description clearly state what the tool does and how it differs from similar tools?

Tool has no description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Tool has no description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

webscraping_ai_htmlD

ParametersJSON Schema

Name	Required	Description	Default
`js`	No	Execute on-page JavaScript using a headless browser (false by default).
`url`	Yes	URL of the target page.
`proxy`	No	Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page.	datacenter
`device`	No	Type of device emulation.
`format`	No	Response format (json or text).
`country`	No	Country of the proxy to use (US by default).
`timeout`	No	Maximum web page retrieval time in ms (20000 by default, maximum is 30000).
`wait_for`	No	CSS selector to wait for before returning the page content.
`js_script`	No	Custom JavaScript code to execute on the target page.
`js_timeout`	No	Maximum JavaScript rendering time in ms (3000 by default).
`custom_proxy`	No	Your own proxy URL in "http://user:password@host:port" format.
`error_on_404`	No	Return error on 404 HTTP status on the target page (false by default).
`error_on_redirect`	No	Return error on redirect on the target page (false by default).
`return_script_result`	No	Return result of the custom JavaScript code execution.

TDQS

D1/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Tool has no description.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness1/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Tool has no description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Tool has no description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Tool has no description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose1/5

Does the description clearly state what the tool does and how it differs from similar tools?

Tool has no description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Tool has no description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

webscraping_ai_questionD

ParametersJSON Schema

Name	Required	Description	Default
`js`	No	Execute on-page JavaScript using a headless browser (false by default).
`url`	Yes	URL of the target page.
`proxy`	No	Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page.	datacenter
`device`	No	Type of device emulation.
`country`	No	Country of the proxy to use (US by default).
`timeout`	No	Maximum web page retrieval time in ms (20000 by default, maximum is 30000).
`question`	Yes	Question or instructions to ask the LLM model about the target page.
`wait_for`	No	CSS selector to wait for before returning the page content.
`js_script`	No	Custom JavaScript code to execute on the target page.
`js_timeout`	No	Maximum JavaScript rendering time in ms (3000 by default).
`custom_proxy`	No	Your own proxy URL in "http://user:password@host:port" format.
`error_on_404`	No	Return error on 404 HTTP status on the target page (false by default).
`error_on_redirect`	No	Return error on redirect on the target page (false by default).

TDQS

D1/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Tool has no description.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness1/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Tool has no description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Tool has no description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Tool has no description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose1/5

Does the description clearly state what the tool does and how it differs from similar tools?

Tool has no description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Tool has no description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

webscraping_ai_selectedD

ParametersJSON Schema

Name	Required	Description	Default
`js`	No	Execute on-page JavaScript using a headless browser (false by default).
`url`	Yes	URL of the target page.
`proxy`	No	Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page.	datacenter
`device`	No	Type of device emulation.
`format`	No	Response format (json or text).	json
`country`	No	Country of the proxy to use (US by default).
`timeout`	No	Maximum web page retrieval time in ms (20000 by default, maximum is 30000).
`selector`	Yes	CSS selector to extract content for.
`wait_for`	No	CSS selector to wait for before returning the page content.
`js_script`	No	Custom JavaScript code to execute on the target page.
`js_timeout`	No	Maximum JavaScript rendering time in ms (3000 by default).
`custom_proxy`	No	Your own proxy URL in "http://user:password@host:port" format.
`error_on_404`	No	Return error on 404 HTTP status on the target page (false by default).
`error_on_redirect`	No	Return error on redirect on the target page (false by default).

TDQS

D1/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Tool has no description.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness1/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Tool has no description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Tool has no description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Tool has no description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose1/5

Does the description clearly state what the tool does and how it differs from similar tools?

Tool has no description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Tool has no description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

webscraping_ai_selected_multipleD

ParametersJSON Schema

Name	Required	Description	Default
`js`	No	Execute on-page JavaScript using a headless browser (false by default).
`url`	Yes	URL of the target page.
`proxy`	No	Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page.	datacenter
`device`	No	Type of device emulation.
`country`	No	Country of the proxy to use (US by default).
`timeout`	No	Maximum web page retrieval time in ms (20000 by default, maximum is 30000).
`wait_for`	No	CSS selector to wait for before returning the page content.
`js_script`	No	Custom JavaScript code to execute on the target page.
`selectors`	Yes	Array of CSS selectors to extract content for.
`js_timeout`	No	Maximum JavaScript rendering time in ms (3000 by default).
`custom_proxy`	No	Your own proxy URL in "http://user:password@host:port" format.
`error_on_404`	No	Return error on 404 HTTP status on the target page (false by default).
`error_on_redirect`	No	Return error on redirect on the target page (false by default).

TDQS

D1/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Tool has no description.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness1/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Tool has no description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Tool has no description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Tool has no description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose1/5

Does the description clearly state what the tool does and how it differs from similar tools?

Tool has no description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Tool has no description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

webscraping_ai_textD

ParametersJSON Schema

Name	Required	Description	Default
`js`	No	Execute on-page JavaScript using a headless browser (false by default).
`url`	Yes	URL of the target page.
`proxy`	No	Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page.	datacenter
`device`	No	Type of device emulation.
`country`	No	Country of the proxy to use (US by default).
`timeout`	No	Maximum web page retrieval time in ms (20000 by default, maximum is 30000).
`wait_for`	No	CSS selector to wait for before returning the page content.
`js_script`	No	Custom JavaScript code to execute on the target page.
`js_timeout`	No	Maximum JavaScript rendering time in ms (3000 by default).
`text_format`	No	Format of the text response.	json
`custom_proxy`	No	Your own proxy URL in "http://user:password@host:port" format.
`error_on_404`	No	Return error on 404 HTTP status on the target page (false by default).
`return_links`	No	Return links from the page body text.
`error_on_redirect`	No	Return error on redirect on the target page (false by default).

TDQS

D1/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Tool has no description.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness1/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Tool has no description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Tool has no description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Tool has no description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose1/5

Does the description clearly state what the tool does and how it differs from similar tools?

Tool has no description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Tool has no description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

6 tool updatesv1.0.5Jun 24, 2026

Changedwebscraping_ai_fields1 field changed

changedInput schema / properties / country / enum

Previous value: -[
-  "us",
-  "gb",
-  "de",
-  "it",
-  "fr",
-  "ca",
-  "es",
-  "ru",
-  "jp",
-  "kr",
-  "in"
-]New value: +[
+  "us",
+  "gb",
+  "de",
+  "it",
+  "fr",
+  "ca",
+  "es",
+  "ru",
+  "jp",
+  "kr",
+  "in",
+  "hk",
+  "tr"
+]

Changedwebscraping_ai_html1 field changed

changedInput schema / properties / country / enum

Previous value: -[
-  "us",
-  "gb",
-  "de",
-  "it",
-  "fr",
-  "ca",
-  "es",
-  "ru",
-  "jp",
-  "kr",
-  "in"
-]New value: +[
+  "us",
+  "gb",
+  "de",
+  "it",
+  "fr",
+  "ca",
+  "es",
+  "ru",
+  "jp",
+  "kr",
+  "in",
+  "hk",
+  "tr"
+]

Changedwebscraping_ai_question1 field changed

changedInput schema / properties / country / enum

Previous value: -[
-  "us",
-  "gb",
-  "de",
-  "it",
-  "fr",
-  "ca",
-  "es",
-  "ru",
-  "jp",
-  "kr",
-  "in"
-]New value: +[
+  "us",
+  "gb",
+  "de",
+  "it",
+  "fr",
+  "ca",
+  "es",
+  "ru",
+  "jp",
+  "kr",
+  "in",
+  "hk",
+  "tr"
+]

Changedwebscraping_ai_selected1 field changed

changedInput schema / properties / country / enum

Previous value: -[
-  "us",
-  "gb",
-  "de",
-  "it",
-  "fr",
-  "ca",
-  "es",
-  "ru",
-  "jp",
-  "kr",
-  "in"
-]New value: +[
+  "us",
+  "gb",
+  "de",
+  "it",
+  "fr",
+  "ca",
+  "es",
+  "ru",
+  "jp",
+  "kr",
+  "in",
+  "hk",
+  "tr"
+]

Changedwebscraping_ai_selected_multiple1 field changed

changedInput schema / properties / country / enum

Previous value: -[
-  "us",
-  "gb",
-  "de",
-  "it",
-  "fr",
-  "ca",
-  "es",
-  "ru",
-  "jp",
-  "kr",
-  "in"
-]New value: +[
+  "us",
+  "gb",
+  "de",
+  "it",
+  "fr",
+  "ca",
+  "es",
+  "ru",
+  "jp",
+  "kr",
+  "in",
+  "hk",
+  "tr"
+]

Changedwebscraping_ai_text1 field changed

changedInput schema / properties / country / enum

Previous value: -[
-  "us",
-  "gb",
-  "de",
-  "it",
-  "fr",
-  "ca",
-  "es",
-  "ru",
-  "jp",
-  "kr",
-  "in"
-]New value: +[
+  "us",
+  "gb",
+  "de",
+  "it",
+  "fr",
+  "ca",
+  "es",
+  "ru",
+  "jp",
+  "kr",
+  "in",
+  "hk",
+  "tr"
+]

6 tool updatesv1.0.4May 17, 2026

Changedwebscraping_ai_fields2 fields changed

changedInput schema / properties / proxy / description

Previous value: -"Type of proxy, datacenter or residential (datacenter by default)."New value: +"Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page."

changedInput schema / properties / proxy / enum

Previous value: -[
-  "datacenter",
-  "residential"
-]New value: +[
+  "datacenter",
+  "residential",
+  "stealth"
+]

Changedwebscraping_ai_html2 fields changed

changedInput schema / properties / proxy / description

Previous value: -"Type of proxy, datacenter or residential (datacenter by default)."New value: +"Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page."

changedInput schema / properties / proxy / enum

Previous value: -[
-  "datacenter",
-  "residential"
-]New value: +[
+  "datacenter",
+  "residential",
+  "stealth"
+]

Changedwebscraping_ai_question2 fields changed

changedInput schema / properties / proxy / description

Previous value: -"Type of proxy, datacenter or residential (datacenter by default)."New value: +"Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page."

changedInput schema / properties / proxy / enum

Previous value: -[
-  "datacenter",
-  "residential"
-]New value: +[
+  "datacenter",
+  "residential",
+  "stealth"
+]

Changedwebscraping_ai_selected2 fields changed

changedInput schema / properties / proxy / description

Previous value: -"Type of proxy, datacenter or residential (datacenter by default)."New value: +"Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page."

changedInput schema / properties / proxy / enum

Previous value: -[
-  "datacenter",
-  "residential"
-]New value: +[
+  "datacenter",
+  "residential",
+  "stealth"
+]

Changedwebscraping_ai_selected_multiple2 fields changed

changedInput schema / properties / proxy / description

Previous value: -"Type of proxy, datacenter or residential (datacenter by default)."New value: +"Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page."

changedInput schema / properties / proxy / enum

Previous value: -[
-  "datacenter",
-  "residential"
-]New value: +[
+  "datacenter",
+  "residential",
+  "stealth"
+]

Changedwebscraping_ai_text2 fields changed

changedInput schema / properties / proxy / description

Previous value: -"Type of proxy, datacenter or residential (datacenter by default)."New value: +"Type of proxy: datacenter, residential, or stealth (datacenter by default). Use residential if the site restricts datacenter traffic, or stealth for the most heavily protected sites with advanced anti-bot detection. Residential and stealth requests cost more than datacenter — see the pricing page."

changedInput schema / properties / proxy / enum

Previous value: -[
-  "datacenter",
-  "residential"
-]New value: +[
+  "datacenter",
+  "residential",
+  "stealth"
+]

7 tool updatesv1.0.2May 3, 2026
- Addedwebscraping_ai_account
- Addedwebscraping_ai_fields
- Addedwebscraping_ai_html
- Addedwebscraping_ai_question
- Addedwebscraping_ai_selected
- Addedwebscraping_ai_selected_multiple
- Addedwebscraping_ai_text

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

D1.8/5.0

Scored 2026-09-10 02:05 across 7 tools

Disambiguation3/5

The names suggest distinct output modes, but several tools overlap conceptually (html vs. text vs. fields vs. question), and selected vs. selected_multiple could easily confuse an agent without descriptions. The lack of descriptions worsens this ambiguity.

Naming Consistency4/5

All tools share the webscraping_ai_ prefix and snake_case convention, making the set look cohesive. The suffixes are mostly nouns/adjectives rather than verb_noun actions, but the pattern is predictable and consistent.

Tool Count4/5

Seven tools is a reasonable size for a scraping-focused serverched. The count covers several output modes without feeling bloated, though selected and selected_multiple could arguably be one parameterized tool.

Completeness4/5

The domain is inferable from the names: HTML/text/field extraction, selected elements, and question-answering over scraped content. It lacks some common extras like link extraction or screenshot, but the core scraping workflow appears covered.

Maintenance

ActivitySlowing

ResponsivenessNo issues

Resources

Need Help?

Reddit Discussion

Related Servers

Related MCP Connectors

scrapeunblocker-mcp-remote A
com.scrapeunblocker
Fetch any web page's HTML, AI-parsed JSON, or Google results via the ScrapeUnblocker anti-bot API
Web Scraping Browser Automation
42
oxylabs-mcp B
io.oxylabs
Fetch and process content from specified URLs & sources using the Oxylabs Web Scraper API.
Web Scraping Browser Automation
101105
web-access AOAuth
ai.usestring
The most accurate web access API. Stop getting blocked.
Browser Automation Web Scraping
22
WyndPath Web DataOAuth
com.wyndpath
Fetch and extract data from any public web page, even JS-rendered or anti-bot protected
Browser Automation Web Scraping
1

Related MCP Servers

FreeCrawl MCP Server
Web Scraping Browser Automation Documentation Access
dylan-gluck
A
license
Not graded
quality
D
maintenance
Enables web scraping and document processing with JavaScript execution, anti-detection measures, batch processing, and structured data extraction. Supports multiple formats including markdown, HTML, screenshots, and handles PDFs with OCR capabilities.
Updated a year ago (2025-08-20 03:34 UTC)
4
MIT
Scrapfly MCP
Web Scraping
scrapfly
A
license
A
quality
B
maintenance
Gives AI assistants real-time access to web data with the ability to scrape any website, bypass anti-bot systems, extract structured data, and capture screenshots through enterprise-grade infrastructure.
Updated 4 days ago (2026-09-12 03:43 UTC)
2
6
65 npm
14
ISC
alterlab-mcp-server
Web Scraping Browser Automation
RapierCraft
A
license
A
quality
A
maintenance
Enables web scraping, structured data extraction, and screenshot capture with automatic anti-bot bypass, supporting JavaScript rendering, proxy rotation, and tiered pricing.
Updated 12 days ago (2026-09-04 07:13 UTC)
25
98 npm
1
MIT
scraper-api-mcp
bitterdev
A
license
B
quality
B
maintenance
Enables AI assistants to access structured public web data (profiles, posts, videos, etc.) from social networks and directories via natural language, by forwarding tool calls to the scraper-api.com API.
Updated 2 months ago (2026-07-23 09:29 UTC)
84
6 npm
MIT

WebScraping.AI MCP 서버

특징

설치

npx로 실행

수동 설치

커서에서 구성

Claude Desktop에서 실행

구성

환경 변수

필수의

선택적 구성

구성 예제

사용 가능한 도구

1. 질문 도구( webscraping_ai_question )

2. 필드 도구( webscraping_ai_fields )

3. HTML 도구( webscraping_ai_html )

4. 텍스트 도구( webscraping_ai_text )

5. 선택된 도구( webscraping_ai_selected )

6. 선택된 다중 도구( webscraping_ai_selected_multiple )

7. 계정 도구( webscraping_ai_account )

모든 도구에 대한 공통 옵션

오류 처리

LLM과의 통합

예: MCP를 사용하여 Claude 구성

개발

기여하다

특허

Available Tools

Tool Schema Changelog

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

Appeared in Searches

1. 질문 도구( `webscraping_ai_question` )

2. 필드 도구( `webscraping_ai_fields` )

3. HTML 도구( `webscraping_ai_html` )

4. 텍스트 도구( `webscraping_ai_text` )

5. 선택된 도구( `webscraping_ai_selected` )

6. 선택된 다중 도구( `webscraping_ai_selected_multiple` )

7. 계정 도구( `webscraping_ai_account` )