Skip to main content
Glama

HasData

by com.hasdata
OverviewDiscussionsAdmin

web_scraping_web_scraping: POST /

hasdata_web_scraping_web_scraping_scrapeWebPage
Destructive

Scrape Web Page

Universal web scraper that fetches any public URL through managed proxies (datacenter or residential, geo-targeted) with optional JS rendering, custom headers, wait conditions, jsScenario actions (click, scroll, fill, waitFor), screenshots, resource/ad/URL blocking (these require JS rendering), and extractRules/aiExtractRules for LLM-driven structured extraction. Returns HTML, text, markdown, and/or JSON along with status code, extracted emails and links, CSS-selector extractions, and AI-structured fields per schema. Use as a fallback/universal fetcher for sites without a dedicated API, for scraping JS-heavy SPAs, capturing screenshots, or producing clean markdown/structured JSON to feed downstream parsers, RAG pipelines, or data warehouses.

Input Schema

NameRequiredDescriptionDefault
urlYesThe URL of the web page to scrape.
waitNoTime in milliseconds to wait after the page load.
headersNoOptional custom headers to send with the request.
waitForNoCSS selector to wait for before scraping.
autoModeNoUse the cheapest proxy type and JS rendering that load the page, overriding `jsRendering` and `proxyType`. The chosen values are returned in `requestMetadata` and billed.
blockAdsNoWhether to block ads.
blockUrlsNoList of URLs to block.
proxyTypeNoType of proxy to use.
jsScenarioNoEnables custom JavaScript interactions on the target webpage during scraping. It's an array where each object defines a specific action or step. These actions can include clicking elements, waiting for elements, executing custom scripts, and more. Key actions within this field include: - `evaluate`: Run custom JavaScript code on the page. - `click`: Click on an element specified by a CSS selector. - `wait`: Pause for a set duration (in milliseconds). - `waitFor`: Delay until a specific element appears. - `waitForAndClick`: Combine waiting for an element and then clicking it. - `scrollX`, `scrollY`: Scroll to specified positions on the page. - `fill`: Enter values into input fields identified by CSS selectors. Actions are executed sequentially.
screenshotNoWhether to take a screenshot of the page.
excludeTagsNoThe `excludeTags` parameter accepts an array of valid CSS selectors. Elements matching these selectors will be removed from the final output. Each value must be a valid `querySelectorAll` selector. This can be used to remove ads, scripts, or other unwanted sections.
jsRenderingNoEnable JavaScript rendering.
extractLinksNoExtract links from the page.
extractRulesNoRules for extracting specific data from the page. For example: `{ "title": "h1", "link_href": "a#link @href", "page_text": "body" }`
outputFormatNoThe outputFormat parameter specifies the desired response format: `html`, `text`, `markdown`, or `json`. If only one of `html`, `text`, or `markdown` is provided, the API returns the response in that format. If multiple formats are specified, the API returns a JSON response with keys for each requested format. If `json` is included with any other format, the API returns a JSON response with keys for the other specified formats.
proxyCountryNoOptional proxy country code.
extractEmailsNoExtract emails from the page.
aiExtractRulesNoDefines custom rules for AI-based data extraction using LLMs. This enables the system to extract structured data directly from the HTML of the page. Each key in the object represents a desired output field name, and the value specifies its type and optional description to guide the AI. Supported types: - `string`: plain text value - `number`: numeric value - `boolean`: true/false - `list`: an array of values - `item`: a nested object with its own structure defined under `output`
blockResourcesNoWhether to block loading of resources like images and stylesheets.
includeOnlyTagsNoThe `includeOnlyTags` parameter accepts an array of valid CSS selectors. When specified, only the elements matching these selectors will be included in the response content. Each value must be a valid `querySelectorAll` selector. Useful for extracting specific parts of the document.
removeBase64ImagesNoIf set to `true`, any images embedded as base64-encoded strings will be removed from the output. Useful for reducing response size or when base64 images are not needed.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / properties / autoMode
      Added value: +{
+  "description": "Use the cheapest proxy type and JS rendering that load the page, overriding `jsRendering` and `proxyType`. The chosen values are returned in `requestMetadata` and billed.",
+  "type": "boolean"
+}
  2. First observed

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnlyHint=false, openWorldHint=true, destructiveHint=true), the description discloses meaningful behavior: proxy management and geo-targeting, the requirement that blocking features need JS rendering, and billing implications (autoMode is billed and chosen values are returned in requestMetadata). It doesn't elaborate on the destructive/open-world side effects flagged by annotations, but adds real operational context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose and when-to-use guidance are front-loaded, and the size is justified by the 21-parameter surface. However, the core content is delivered as one very dense compound sentence, which is information-rich but harder to scan than a structured breakdown would be.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the return-value burden and does so well: 'Returns HTML, text, markdown, and/or JSON along with status code, extracted emails and links, CSS-selector extractions, and AI-structured fields per schema.' Combined with the feature enumeration, an agent has everything needed to call this complex tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds conceptual framing above the schema by grouping capabilities (jsScenario actions like click/scroll/fill/waitFor, extractRules vs aiExtractRules for LLM extraction) and clarifying that output is HTML/text/markdown/JSON, which helps an agent reason about parameter combinations.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb (scrape/fetch) and resource (any public URL via managed proxies) and enumerates the distinct capabilities: JS rendering, custom headers, wait conditions, jsScenario actions, screenshots, blocking, and extractRules/aiExtractRules. No siblings exist, so no differentiation is needed, and an agent can immediately tell this is the universal fetcher.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly routes usage: 'Use as a fallback/universal fetcher for sites without a dedicated API, for scraping JS-heavy SPAs, capturing screenshots, or producing clean markdown/structured JSON.' This gives clear when-to-use conditions, though it states no explicit exclusions or named alternatives (none exist among siblings).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All HasData toolsInspect
Try in Browser
Let your AI agent see the webWeb-to-screenshot MCP for Claude, Cursor and Codex. No cookie banners. Free to start.snap-render.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources