PyScrappy

Overview Schema Source Related Servers Discussions Admin

Scrape Url

scrape_url

Extract structured data from any URL: text, links, images, tables, and metadata. Handle dynamic pages with JS rendering, pagination, or CSS selectors.

Instructions

Scrape any HTTP(S) URL and return a ScrapeToolResult whose data holds one object per page containing extracted text (with word_count), links, images, tables, and page metadata.

Fetches the page over the network and parses the HTML; no data is stored or mutated. By default it makes a plain static HTTP request, so pages built client-side with JavaScript come back nearly empty. When that is detected, the returned errors list gets a hint to retry with render_js=true; set render_js=true to render with a headless browser instead (requires the pyscrappy[browser] extra). On empty or failed results, data is [], count is 0, and errors describes the problem rather than raising.

Input Schema

TableJSON Schema

Name	Required	Description
`url`	Yes	String, the page URL to scrape including scheme, e.g. "https://example.com/products". Required, no default.
`max_pages`	No	Integer, follow "next"-style pagination up to this many pages, e.g. 3. Default 1 (scrape only the given URL).
`render_js`	No	Boolean, render JavaScript with a headless browser backend, e.g. True. Default False; allowed values True or False, and True needs the pyscrappy[browser] extra installed.
`selectors`	No	Optional dict mapping output field name to CSS selector to extract specific values into each data item, e.g. {"title": "h1", "price": ".amount"}. Default None (returns only the standard text/links/images/tables/metadata).

Output Schema

TableJSON Schema

Name	Required	Description	Default
`data`	No
`count`	No
`errors`	No
`scraper`	No
`source_urls`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed4 schema fields changedJul 31, 2026

changedInput schema / properties / max_pages / description

Previous value: -"Follow pagination up to this many pages (default 1)."New value: +"Integer, follow \"next\"-style pagination up to this many pages, e.g. 3. Default 1 (scrape only the given URL)."

changedInput schema / properties / render_js / description

Previous value: -"Render JavaScript with a browser backend (needs pyscrappy[browser])."New value: +"Boolean, render JavaScript with a headless browser backend, e.g. True. Default False; allowed values True or False, and True needs the pyscrappy[browser] extra installed."

changedInput schema / properties / selectors / description

Previous value: -"Optional CSS selectors, e.g. {\"title\": \"h1\", \"price\": \".amount\"}."New value: +"Optional dict mapping output field name to CSS selector to extract specific values into each data item, e.g. {\"title\": \"h1\", \"price\": \".amount\"}. Default None (returns only the standard text/links/images/tables/metadata)."

changedInput schema / properties / url / description

Previous value: -"The page to scrape."New value: +"String, the page URL to scrape including scheme, e.g. \"https://example.com/products\". Required, no default."

Changed22 schema fields changedv1.3.3Jul 31, 2026

addedInput schema / additionalProperties
```
Added value: +false
```

addedInput schema / properties / max_pages / description

Added value: +"Follow pagination up to this many pages (default 1)."

removedInput schema / properties / max_pages / title
```
Removed value: -"Max Pages"
```

addedInput schema / properties / render_js / description

Added value: +"Render JavaScript with a browser backend (needs pyscrappy[browser])."

removedInput schema / properties / render_js / title
```
Removed value: -"Render Js"
```

addedInput schema / properties / selectors / description

Added value: +"Optional CSS selectors, e.g. {\"title\": \"h1\", \"price\": \".amount\"}."

removedInput schema / properties / selectors / title
```
Removed value: -"Selectors"
```
addedInput schema / properties / url / description
```
Added value: +"The page to scrape."
```
removedInput schema / properties / url / title
```
Removed value: -"Url"
```
removedInput schema / title
```
Removed value: -"scrape_urlArguments"
```

removedOutput schema / $defs

Removed value: -{
-  "ToolError": {
-    "description": "A non-fatal problem encountered while scraping.",
-    "properties": {
-      "message": {
-        "title": "Message",
-        "type": "string"
-      },
-      "url": {
-        "title": "Url",
-        "type": "string"
-      }
-    },
-    "required": [
-      "url",
-      "message"
-    ],
-    "title": "ToolError",
-    "type": "object"
-  }
-}

removedOutput schema / properties / count / title
```
Removed value: -"Count"
```
removedOutput schema / properties / data / title
```
Removed value: -"Data"
```
removedOutput schema / properties / errors / items / $ref
```
Removed value: -"#/$defs/ToolError"
```

addedOutput schema / properties / errors / items / description

Added value: +"A non-fatal problem encountered while scraping."

addedOutput schema / properties / errors / items / properties

Added value: +{
+  "message": {
+    "type": "string"
+  },
+  "url": {
+    "type": "string"
+  }
+}

addedOutput schema / properties / errors / items / required
```
Added value: +[
+  "url",
+  "message"
+]
```
addedOutput schema / properties / errors / items / type
```
Added value: +"object"
```
removedOutput schema / properties / errors / title
```
Removed value: -"Errors"
```
removedOutput schema / properties / scraper / title
```
Removed value: -"Scraper"
```
removedOutput schema / properties / source_urls / title
```
Removed value: -"Source Urls"
```
removedOutput schema / title
```
Removed value: -"ScrapeToolResult"
```

Addedv1.2.0Jul 26, 2026
RemovedJul 26, 2026
First observedv1.1.6Jul 26, 2026

TDQS

A4.5/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does it thoroughly: network fetch, no storage/mutation, static vs JS rendering behavior, the error hint, and the exact empty-result shape (data [], count 0, errors) instead of raising. This is exemplary disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but efficient: first sentence states purpose and output shape, and the rest covers behavior, failure mode, and the JS-rendering option. Every sentence earns its place; no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the output schema exists and the input schema is fully documented, the description supplies the missing behavioral context: network effects, JS caveat, dependency requirement, and failure semantics. Nothing needed to invoke the tool correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the schema already documents url, max_pages, render_js, and selectors with defaults and examples. The tool description reinforces the render_js behavior but adds little meaning beyond the schema, matching the baseline for high coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Scrape any HTTP(S) URL and return...' which names a specific verb, a clear resource, and the general-purpose scope. This distinguishes it from the specialized scrapers in the sibling list without requiring a tool-by-tool comparison.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives concrete guidance for the main decision point: use the default static request unless the page is client-side rendered, in which case set render_js=true. It does not explicitly call out sibling tools as alternatives or state when not to use this tool, so it stops short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.