Skip to main content
Glama

HasData

by com.hasdata
OverviewDiscussionsAdmin

web_scraping_web_scraping: POST /

hasdata_web_scraping_web_scraping_scrapeWebPage

Scrape Web Page

Universal web scraper that fetches any public URL through managed proxies (datacenter or residential, geo-targeted) with optional JS rendering, custom headers, wait conditions, jsScenario actions (click, scroll, fill, waitFor), screenshots, resource/ad/URL blocking, and extractRules/aiExtractRules for LLM-driven structured extraction. Returns HTML, text, markdown, and/or JSON along with status code, extracted emails and links, CSS-selector extractions, and AI-structured fields per schema. Use as a fallback/universal fetcher for sites without a dedicated API, for scraping JS-heavy SPAs, bypassing bot protections, capturing screenshots, or producing clean markdown/structured JSON to feed downstream parsers, RAG pipelines, or data warehouses.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL of the web page to scrape.
waitNoTime in milliseconds to wait after the page load.
headersNoOptional custom headers to send with the request.
waitForNoCSS selector to wait for before scraping.
blockAdsNoWhether to block ads.
blockUrlsNoList of URLs to block.
proxyTypeNoType of proxy to use.
jsScenarioNoEnables custom JavaScript interactions on the target webpage during scraping. It's an array where each object defines a specific action or step. These actions can include clicking elements, waiting for elements, executing custom scripts, and more. Key actions within this field include: - `evaluate`: Run custom JavaScript code on the page. - `click`: Click on an element specified by a CSS selector. - `wait`: Pause for a set duration (in milliseconds). - `waitFor`: Delay until a specific element appears. - `waitForAndClick`: Combine waiting for an element and then clicking it. - `scrollX`, `scrollY`: Scroll to specified positions on the page. - `fill`: Enter values into input fields identified by CSS selectors. Actions are executed sequentially.
screenshotNoWhether to take a screenshot of the page.
excludeTagsNoThe `excludeTags` parameter accepts an array of valid CSS selectors. Elements matching these selectors will be removed from the final output. Each value must be a valid `querySelectorAll` selector. This can be used to remove ads, scripts, or other unwanted sections.
jsRenderingNoEnable JavaScript rendering.
extractLinksNoExtract links from the page.
extractRulesNoRules for extracting specific data from the page. For example: `{ "title": "h1", "link_href": "a#link @href", "page_text": "body" }`
outputFormatNoThe outputFormat parameter specifies the desired response format: `html`, `text`, `markdown`, or `json`. If only one of `html`, `text`, or `markdown` is provided, the API returns the response in that format. If multiple formats are specified, the API returns a JSON response with keys for each requested format. If `json` is included with any other format, the API returns a JSON response with keys for the other specified formats.
proxyCountryNoOptional proxy country code.
extractEmailsNoExtract emails from the page.
aiExtractRulesNoDefines custom rules for AI-based data extraction using LLMs. This enables the system to extract structured data directly from the HTML of the page. Each key in the object represents a desired output field name, and the value specifies its type and optional description to guide the AI. Supported types: - `string`: plain text value - `number`: numeric value - `boolean`: true/false - `list`: an array of values - `item`: a nested object with its own structure defined under `output`
blockResourcesNoWhether to block loading of resources like images and stylesheets.
includeOnlyTagsNoThe `includeOnlyTags` parameter accepts an array of valid CSS selectors. When specified, only the elements matching these selectors will be included in the response content. Each value must be a valid `querySelectorAll` selector. Useful for extracting specific parts of the document.
removeBase64ImagesNoIf set to `true`, any images embedded as base64-encoded strings will be removed from the output. Useful for reducing response size or when base64 images are not needed.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full transparency burden. It discloses key behaviors such as using managed proxies, optional JS rendering, waiting conditions, blocking ads/resources, screenshots, and extraction capabilities. It does not mention rate limits or legal/ethical caveats, but the core behavioral traits are adequately described.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is reasonably concise for a tool with 20 parameters. It front-loads the core purpose and capabilities, follows with specific use cases, and avoids redundant repetition of schema details. The structure is clear and scannable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (20 parameters, nested objects, no output schema), the description provides sufficient context: it explains what the tool does, when to use it, what output formats are available, and what extraction features exist. It does not describe an output schema, but the description adequately covers expected return types and capabilities.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The narrative description summarizes capabilities but does not add significant per-parameter meaning beyond the already detailed input schema descriptions. The schema itself provides strong parameter documentation, including nested objects and enum values.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool as a universal web scraper that fetches any public URL, with a specific verb ('Scrape Web Page') and a well-defined resource. It also distinguishes itself from the many specialized sibling tools by explicitly positioning itself as a fallback for sites without a dedicated API.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidance: use as a fallback/universal fetcher for sites without a dedicated API, for JS-heavy SPAs, bypassing bot protections, capturing screenshots, or producing clean markdown/structured JSON. This clearly tells an agent when to choose this tool over the specialized siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All HasData toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources