Web Scraping MCP Server

Overview Schema Source Related Servers Score Discussions Admin

web_scraping_web_scraping: POST /

hasdata_web_scraping_web_scraping_scrapeWebPage

Fetch any public URL through managed proxies while rendering JavaScript, then extract clean HTML, markdown, or structured JSON for downstream use.

Instructions

Scrape Web Page

Universal web scraper that fetches any public URL through managed proxies (datacenter or residential, geo-targeted) with optional JS rendering, custom headers, wait conditions, jsScenario actions (click, scroll, fill, waitFor), screenshots, resource/ad/URL blocking, and extractRules/aiExtractRules for LLM-driven structured extraction. Returns HTML, text, markdown, and/or JSON along with status code, extracted emails and links, CSS-selector extractions, and AI-structured fields per schema. Use as a fallback/universal fetcher for sites without a dedicated API, for scraping JS-heavy SPAs, bypassing bot protections, capturing screenshots, or producing clean markdown/structured JSON to feed downstream parsers, RAG pipelines, or data warehouses.

Input Schema

TableJSON Schema

Name	Required	Description
`url`	Yes	The URL of the web page to scrape.
`wait`	No	Time in milliseconds to wait after the page load.
`headers`	No	Optional custom headers to send with the request.
`waitFor`	No	CSS selector to wait for before scraping.
`blockAds`	No	Whether to block ads.
`blockUrls`	No	List of URLs to block.
`proxyType`	No	Type of proxy to use.
`jsScenario`	No	Enables custom JavaScript interactions on the target webpage during scraping. It's an array where each object defines a specific action or step. These actions can include clicking elements, waiting for elements, executing custom scripts, and more. Key actions within this field include: - `evaluate`: Run custom JavaScript code on the page. - `click`: Click on an element specified by a CSS selector. - `wait`: Pause for a set duration (in milliseconds). - `waitFor`: Delay until a specific element appears. - `waitForAndClick`: Combine waiting for an element and then clicking it. - `scrollX`, `scrollY`: Scroll to specified positions on the page. - `fill`: Enter values into input fields identified by CSS selectors. Actions are executed sequentially.
`screenshot`	No	Whether to take a screenshot of the page.
`excludeTags`	No	The `excludeTags` parameter accepts an array of valid CSS selectors. Elements matching these selectors will be removed from the final output. Each value must be a valid `querySelectorAll` selector. This can be used to remove ads, scripts, or other unwanted sections.
`jsRendering`	No	Enable JavaScript rendering.
`extractLinks`	No	Extract links from the page.
`extractRules`	No	Rules for extracting specific data from the page. For example: `{ "title": "h1", "link_href": "a#link @href", "page_text": "body" }`
`outputFormat`	No	The outputFormat parameter specifies the desired response format: `html`, `text`, `markdown`, or `json`. If only one of `html`, `text`, or `markdown` is provided, the API returns the response in that format. If multiple formats are specified, the API returns a JSON response with keys for each requested format. If `json` is included with any other format, the API returns a JSON response with keys for the other specified formats.
`proxyCountry`	No	Optional proxy country code.
`extractEmails`	No	Extract emails from the page.
`aiExtractRules`	No	Defines custom rules for AI-based data extraction using LLMs. This enables the system to extract structured data directly from the HTML of the page. Each key in the object represents a desired output field name, and the value specifies its type and optional description to guide the AI. Supported types: - `string`: plain text value - `number`: numeric value - `boolean`: true/false - `list`: an array of values - `item`: a nested object with its own structure defined under `output`
`blockResources`	No	Whether to block loading of resources like images and stylesheets.
`includeOnlyTags`	No	The `includeOnlyTags` parameter accepts an array of valid CSS selectors. When specified, only the elements matching these selectors will be included in the response content. Each value must be a valid `querySelectorAll` selector. Useful for extracting specific parts of the document.
`removeBase64Images`	No	If set to `true`, any images embedded as base64-encoded strings will be removed from the output. Useful for reducing response size or when base64 images are not needed.

Schema Changelog

Changes observed during successful MCP inspections.

First observedv1.0.0Sep 10, 2026

TDQS

A4.3/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full transparency burden. It discloses key behaviors such as using managed proxies, optional JS rendering, waiting conditions, blocking ads/resources, screenshots, and extraction capabilities. It does not mention rate limits or legal/ethical caveats, but the core behavioral traits are adequately described.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is reasonably concise for a tool with 20 parameters. It front-loads the core purpose and capabilities, follows with specific use cases, and avoids redundant repetition of schema details. The structure is clear and scannable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (20 parameters, nested objects, no output schema), the description provides sufficient context: it explains what the tool does, when to use it, what output formats are available, and what extraction features exist. It does not describe an output schema, but the description adequately covers expected return types and capabilities.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The narrative description summarizes capabilities but does not add significant per-parameter meaning beyond the already detailed input schema descriptions. The schema itself provides strong parameter documentation, including nested objects and enum values.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool as a universal web scraper that fetches any public URL, with a specific verb ('Scrape Web Page') and a well-defined resource. It also distinguishes itself from the many specialized sibling tools by explicitly positioning itself as a fallback for sites without a dedicated API.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidance: use as a fallback/universal fetcher for sites without a dedicated API, for JS-heavy SPAs, bypassing bot protections, capturing screenshots, or producing clean markdown/structured JSON. This clearly tells an agent when to choose this tool over the specialized siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server

Other Tools

hasdata_web_scraping_web_scraping_scrapeWebPageA