Scrapy MCP Server

The Scrapy MCP Server is a robust, enterprise-grade web scraping platform that offers comprehensive data extraction capabilities for commercial use.

Core Scraping Capabilities:

• Multiple scraping methods: HTTP requests, Scrapy framework, Selenium, or Playwright with intelligent method selection

• Concurrent processing: Scrape multiple URLs simultaneously with exponential backoff retry mechanisms

• JavaScript support: Fully render dynamic, JavaScript-heavy websites using complete browser rendering

• Advanced data extraction: Configure flexible extraction rules using simple or advanced selectors, or automatically extract structured data like contact information, social media links, product details, and addresses

• Link extraction: Specialized link extraction with domain filtering and internal/external link options

• Form interaction: Automatically fill and submit various form types including text inputs, checkboxes, and file uploads

Anti-Detection & Performance:

• Stealth techniques: Bypass anti-bot measures using undetected-chromedriver, Playwright stealth, random User-Agent rotation, and proxy support

• Performance optimization: In-memory caching, rate limiting, and intelligent request handling to prevent server overload

• Monitoring tools: Track server metrics including request counts, success rates, cache statistics, and detailed performance monitoring

Enterprise Features:

• Ethical compliance: Check robots.txt files for responsible data collection

• Error handling: Robust error classification and handling mechanisms

• Cache management: Clear scraping results cache and manage server resources

Provides web scraping capabilities using the Scrapy framework for large-scale data extraction, with support for concurrent requests, custom pipelines, and advanced crawling features.

Enables browser automation and JavaScript-heavy website scraping through Selenium WebDriver, with support for form filling, element waiting, and dynamic content extraction.

1. Click on "Install Server".

2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.

3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Scrapy MCP Server scrape the pricing page from example.com and convert it to markdown"

That's it! The server will respond to your query, and you can continue using it as needed.

Here is a step-by-step guide with screenshots.