HasData

web_scraping_web_scraping: POST /

hasdata_web_scraping_web_scraping_scrapeWebPage

Destructive

Scrape Web Page

Universal web scraper that fetches any public URL through managed proxies (datacenter or residential, geo-targeted) with optional JS rendering, custom headers, wait conditions, jsScenario actions (click, scroll, fill, waitFor), screenshots, resource/ad/URL blocking (these require JS rendering), and extractRules/aiExtractRules for LLM-driven structured extraction. Returns HTML, text, markdown, and/or JSON along with status code, extracted emails and links, CSS-selector extractions, and AI-structured fields per schema. Use as a fallback/universal fetcher for sites without a dedicated API, for scraping JS-heavy SPAs, bypassing bot protections, capturing screenshots, or producing clean markdown/structured JSON to feed downstream parsers, RAG pipelines, or data warehouses.

Input Schema

TableJSON Schema

Name	Required	Description
`url`	Yes	The URL of the web page to scrape.
`wait`	No	Time in milliseconds to wait after the page load.
`headers`	No	Optional custom headers to send with the request.
`waitFor`	No	CSS selector to wait for before scraping.
`autoMode`	No	Use the cheapest proxy type and JS rendering that load the page, overriding `jsRendering` and `proxyType`. The chosen values are returned in `requestMetadata` and billed.
`blockAds`	No	Whether to block ads.
`blockUrls`	No	List of URLs to block.
`proxyType`	No	Type of proxy to use.
`jsScenario`	No	Enables custom JavaScript interactions on the target webpage during scraping. It's an array where each object defines a specific action or step. These actions can include clicking elements, waiting for elements, executing custom scripts, and more. Key actions within this field include: - `evaluate`: Run custom JavaScript code on the page. - `click`: Click on an element specified by a CSS selector. - `wait`: Pause for a set duration (in milliseconds). - `waitFor`: Delay until a specific element appears. - `waitForAndClick`: Combine waiting for an element and then clicking it. - `scrollX`, `scrollY`: Scroll to specified positions on the page. - `fill`: Enter values into input fields identified by CSS selectors. Actions are executed sequentially.
`screenshot`	No	Whether to take a screenshot of the page.
`excludeTags`	No	The `excludeTags` parameter accepts an array of valid CSS selectors. Elements matching these selectors will be removed from the final output. Each value must be a valid `querySelectorAll` selector. This can be used to remove ads, scripts, or other unwanted sections.
`jsRendering`	No	Enable JavaScript rendering.
`extractLinks`	No	Extract links from the page.
`extractRules`	No	Rules for extracting specific data from the page. For example: `{ "title": "h1", "link_href": "a#link @href", "page_text": "body" }`
`outputFormat`	No	The outputFormat parameter specifies the desired response format: `html`, `text`, `markdown`, or `json`. If only one of `html`, `text`, or `markdown` is provided, the API returns the response in that format. If multiple formats are specified, the API returns a JSON response with keys for each requested format. If `json` is included with any other format, the API returns a JSON response with keys for the other specified formats.
`proxyCountry`	No	Optional proxy country code.
`extractEmails`	No	Extract emails from the page.
`aiExtractRules`	No	Defines custom rules for AI-based data extraction using LLMs. This enables the system to extract structured data directly from the HTML of the page. Each key in the object represents a desired output field name, and the value specifies its type and optional description to guide the AI. Supported types: - `string`: plain text value - `number`: numeric value - `boolean`: true/false - `list`: an array of values - `item`: a nested object with its own structure defined under `output`
`blockResources`	No	Whether to block loading of resources like images and stylesheets.
`includeOnlyTags`	No	The `includeOnlyTags` parameter accepts an array of valid CSS selectors. When specified, only the elements matching these selectors will be included in the response content. Each value must be a valid `querySelectorAll` selector. Useful for extracting specific parts of the document.
`removeBase64Images`	No	If set to `true`, any images embedded as base64-encoded strings will be removed from the output. Useful for reducing response size or when base64 images are not needed.

Schema Changelog

Changes observed during successful MCP inspections.

Changed1 schema field changedOct 7, 2026

addedInput schema / properties / autoMode

Added value: +{
+  "description": "Use the cheapest proxy type and JS rendering that load the page, overriding `jsRendering` and `proxyType`. The chosen values are returned in `requestMetadata` and billed.",
+  "type": "boolean"
+}

First observedApr 28, 2026

TDQS

A4.4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, openWorldHint=true and destructiveHint=true, so the safety bar is lower. The description adds genuinely useful behavioral context beyond that: managed/geo-targeted proxies, that blocking features 'require JS rendering', and that output includes status code plus extracted emails/links/AI fields. It does not reconcile the destructiveHint, nor mention auth or proxy billing (billing is only hinted at in the schema's autoMode).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core capability and then usage guidance; every clause carries information and nothing is redundant filler. The opening sentence is an unusually dense run-on of feature lists, which slightly hurts scannability, but no sentence fails to earn its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 21-parameter, nested-object tool with no output schema, the description compensates well by enumerating the returned artifacts (HTML/text/markdown/JSON, status code, emails, links, CSS extractions, AI fields) and the major optional feature groups. Given the schema's full parameter coverage, an agent has enough to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3 and the schema carries the parameter detail. The description still adds cross-parameter relationships not obvious from the schema alone, notably that resource/ad/URL blocking depends on JS rendering, and it groups the jsScenario action vocabulary (click, scroll, fill, waitFor) into one place for quick comprehension.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Scrape Web Page', 'Universal web scraper that fetches any public URL') and immediately scopes the capability set (proxies, JS rendering, screenshots, extraction). An agent knows exactly what the tool does without opening the schema. No siblings exist to differentiate from, so the remaining burden is met.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The final sentence gives explicit when-to-use contexts: 'fallback/universal fetcher for sites without a dedicated API', 'scraping JS-heavy SPAs', 'bypassing bot protections', 'capturing screenshots', and feeding RAG/warehouse pipelines. It lacks any when-not-to-use or exclusion criteria, which keeps it short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All HasData tools Inspect

Try in Browser

Let your AI agent see the webWeb-to-screenshot MCP for Claude, Cursor and Codex. No cookie banners. Free to start.snap-render.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?