scrape_multiple_webpages
Concurrently scrape multiple URLs using the Scrapy MCP Server. Extract data efficiently with configurable methods (auto, simple, Scrapy, Selenium) and handling for complex web structures.
Instructions
Scrape multiple webpages concurrently.
This tool allows you to scrape multiple URLs at once, which is much faster than scraping them one by one. All URLs will be processed concurrently.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes |
Input Schema (JSON Schema)
{
"$defs": {
"MultipleScrapeRequest": {
"description": "Request model for scraping multiple URLs.",
"properties": {
"extract_config": {
"anyOf": [
{
"additionalProperties": true,
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"description": "Configuration for data extraction",
"title": "Extract Config"
},
"method": {
"default": "auto",
"description": "Scraping method: auto, simple, scrapy, selenium",
"title": "Method",
"type": "string"
},
"urls": {
"description": "List of URLs to scrape",
"items": {
"type": "string"
},
"title": "Urls",
"type": "array"
}
},
"required": [
"urls"
],
"title": "MultipleScrapeRequest",
"type": "object"
}
},
"properties": {
"request": {
"$ref": "#/$defs/MultipleScrapeRequest",
"title": "Request"
}
},
"required": [
"request"
],
"type": "object"
}