scrape_url
Scrape any HTTP(S) URL and extract text, links, images, tables, and page metadata into structured JSON. Supports pagination and optional JavaScript rendering for dynamic pages.
Instructions
Scrape any HTTP(S) URL and return a ScrapeToolResult whose data holds one object per page containing extracted text (with word_count), links, images, tables, and page metadata.
Fetches the page over the network and parses the HTML; no data is stored or mutated. By default it makes a plain static HTTP request, so pages built client-side with JavaScript come back nearly empty. When that is detected, the returned errors list gets a hint to retry with render_js=true; set render_js=true to render with a headless browser instead (requires the pyscrappy[browser] extra). On empty or failed results, data is [], count is 0, and errors describes the problem rather than raising.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | String, the page URL to scrape including scheme, e.g. "https://example.com/products". Required, no default. | |
| max_pages | No | Integer, follow "next"-style pagination up to this many pages, e.g. 3. Default 1 (scrape only the given URL). | |
| render_js | No | Boolean, render JavaScript with a headless browser backend, e.g. True. Default False; allowed values True or False, and True needs the pyscrappy[browser] extra installed. | |
| selectors | No | Optional dict mapping output field name to CSS selector to extract specific values into each data item, e.g. {"title": "h1", "price": ".amount"}. Default None (returns only the standard text/links/images/tables/metadata). |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| data | No | ||
| count | No | ||
| errors | No | ||
| scraper | No | ||
| source_urls | No |