This MCP server enables AI assistants to retrieve text content from bot-protected websites and extract specific information using regex patterns.
Core Capabilities:
Web Page Fetching: Retrieve complete web pages with pagination support, optimized for text-based documentation and reference materials
Pattern Extraction: Search and extract specific content using regular expressions with configurable context around matches
Bot Detection Bypass: Three protection modes (basic, stealth, max-stealth) that automatically escalate when sites block access
Flexible Output: Content delivered in HTML or Markdown format with configurable length limits and continuation from specific positions
Intelligent Integration: Claude automatically selects appropriate tools based on natural language requests without requiring technical commands
Primarily designed for low-volume retrieval of documentation, articles, and reference materials from websites that implement bot detection.
Enables installation of the MCP server through PyPI's package repository, with version tracking and dependency management.
scrapling-fetch-mcp
An MCP server that helps AI assistants access text content from websites that implement bot detection, bridging the gap between what you can see in your browser and what the AI can access.
Intended Use
This tool is optimized for low-volume retrieval of documentation and reference materials (text/HTML only) from websites that implement bot detection. It has not been designed or tested for general-purpose site scraping or data harvesting.
Note: This project was developed in collaboration with Claude Sonnets 3.7 and 4.5, using LLM Context.
Related MCP server: browser-use MCP Server
Installation
Requirements
Python 3.10+
uv package manager
Install
Important: The browser installation downloads hundreds of MB of data and must complete before first use. If the MCP server times out on first use, the browsers may still be installing in the background. Wait a few minutes and try again.
Setup with Claude Desktop
Add this configuration to your Claude Desktop MCP settings:
MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
After updating the config, restart Claude Desktop.
What It Does
This MCP server provides two tools that Claude can use automatically when you ask it to fetch web content:
Page fetching: Retrieves complete web pages with support for pagination
Pattern extraction: Finds and extracts specific content using regex patterns
The AI decides which tool to use based on your request. You just ask naturally:
Protection Modes
The tools support three levels of bot detection bypass:
basic: Fast (1-2s), works for most sites
stealth: Moderate (3-8s), handles more protection
max-stealth: Maximum (10+s), for heavily protected sites
Claude automatically starts with basic mode and escalates if needed.
Tips for Best Results
Just ask naturally - Claude handles the technical details
For large pages, Claude can page through content automatically
For specific searches, mention what you're looking for and Claude will use pattern matching
The metadata returned helps Claude decide whether to page or search
Limitations
Designed for text content only (documentation, articles, references)
Not for high-volume scraping or data harvesting
May not work with sites requiring authentication
Performance varies by site complexity and protection level
Built with Scrapling for web scraping with bot detection bypass.
License
Apache 2.0