scrape_webpage

Extract and scrape webpage content using auto, simple, Scrapy, or Selenium methods. Define extraction rules or wait for specific elements to retrieve targeted data.

Instructions

Scrape a single webpage and extract its content.

This tool can scrape web pages using different methods:

auto: Automatically choose the best method
simple: Fast HTTP requests (no JavaScript)
scrapy: Robust scraping with Scrapy framework
selenium: Full browser rendering (supports JavaScript)

You can specify extraction rules to get specific data from the page.

Input Schema

Name	Required	Description	Default
`request`	Yes

Input Schema (JSON Schema)

{
  "$defs": {
    "ScrapeRequest": {
      "description": "Request model for scraping operations.",
      "properties": {
        "extract_config": {
          "anyOf": [
            {
              "additionalProperties": true,
              "type": "object"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Configuration for data extraction",
          "title": "Extract Config"
        },
        "method": {
          "default": "auto",
          "description": "Scraping method: auto, simple, scrapy, selenium",
          "title": "Method",
          "type": "string"
        },
        "url": {
          "description": "URL to scrape",
          "title": "Url",
          "type": "string"
        },
        "wait_for_element": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "CSS selector to wait for (Selenium only)",
          "title": "Wait For Element"
        }
      },
      "required": [
        "url"
      ],
      "title": "ScrapeRequest",
      "type": "object"
    }
  },
  "properties": {
    "request": {
      "$ref": "#/$defs/ScrapeRequest",
      "title": "Request"
    }
  },
  "required": [
    "request"
  ],
  "type": "object"
}

Scrapy MCP Server

scrape_webpage

Instructions

Input Schema

Input Schema (JSON Schema)

Other Tools from Scrapy MCP Server

Related Tools

MCP directory API