The WebSurfer MCP server enables AI assistants to securely fetch and extract clean, readable text from web pages through a standardized interface.
Fetch web content: Retrieve plain-text content from any publicly accessible HTTP/HTTPS URL using the
search_urltoolExtract clean text: Automatically remove boilerplate elements (navigation, headers, scripts) using trafilatura and BeautifulSoup4 to provide high-quality, readable content
Enhance security: Prevent SSRF attacks by blocking access to private IPs, loopback addresses, reserved IP ranges, and non-HTTP/HTTPS schemes
Manage resources: Enforce content size limits (default 10MB), configurable request timeouts (1-60 seconds, default 10 seconds), and built-in rate limiting
Handle errors gracefully: Receive detailed feedback for network issues, HTTP errors, and content parsing failures
Customize behavior: Configure timeout, user agent, and content limits via environment variables
Integrate seamlessly: Work with MCP-compliant clients like Claude Desktop to enable web browsing capabilities for AI assistants
Used for making async HTTP requests to web pages, enabling efficient fetching of web content
Blocks javascript: URL schemes as part of the security features to prevent potential security vulnerabilities
Built with modern Python async patterns for high performance, requiring Python 3.12 or higher to run
Supports processing XML content types, allowing extraction of text from XML-based web pages
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@WebSurfer MCPget the main content from https://news.ycombinator.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
WebSurfer MCP
WebSurfer is a Model Context Protocol (MCP) server designed to provide Large Language Models (LLMs) with secure and efficient access to web content. It enables AI assistants to fetch, parse, and extract clean text from web pages through a standardized interface.
Core Features
Advanced URL Validation: Implements strict security controls using the
ipaddressmodule to block access to private, loopback, and reserved IP ranges (SSRF protection).Optimized Content Extraction: Utilizes
trafilaturaandBeautifulSoup4to extract high-quality, readable text from HTML, effectively removing boilerplate such as navigation, headers, and scripts.Resource Management: Enforces strict content size limits and request timeouts to ensure system stability and performance.
Rate Limiting: Built-in request throttling to prevent service abuse and manage resource consumption.
Robust Error Handling: Provides granular feedback for network issues, HTTP errors, and content parsing failures.
Related MCP server: MCP Toolkit
System Architecture
The project is composed of several specialized components:
MCPURLSearchServer: The primary server implementation that handles the MCP lifecycle and tool registration.
TextExtractor: Manages asynchronous HTTP sessions and content parsing logic.
URLValidator: Performs security auditing and normalization on input URLs.
Config: Centralizes configuration management via environment variables.
Installation
Prerequisites
Python 3.12 or higher
uv package manager
Setup
Clone the repository:
git clone https://github.com/crybo-rybo/websurfer-mcp cd websurfer-mcpInstall dependencies:
uv sync
Usage
Server Execution
The server communicates via standard I/O (stdio) and is compatible with any MCP-compliant client.
Manual Testing
You can verify the extraction functionality directly from the command line:
Desktop Client Integration
Claude Desktop
To use WebSurfer MCP with Claude Desktop, add the following configuration to your claude_desktop_config.json file.
Path locations:
macOS:
~/Library/Application Support/Claude/claude_desktop_config.jsonWindows:
%APPDATA%\Claude\claude_desktop_config.json
Configuration:
Replace /path/to/websurfer-mcp with the absolute path to your cloned repository.
After updating the configuration, restart Claude Desktop to enable the search_url tool.
Configuration
The server can be configured using the following environment variables:
Variable | Default | Description |
|
| Default request timeout in seconds. |
|
| Maximum allowed timeout in seconds. |
|
| User-Agent string for outgoing requests. |
|
| Maximum content size in bytes (default 10MB). |
Testing
The project maintains a comprehensive test suite covering unit and integration scenarios.
Execute All Tests
Component Testing
Security
WebSurfer MCP is designed with security as a primary concern. It explicitly blocks:
Private IP ranges (e.g., 10.0.0.0/8, 192.168.0.0/16)
Loopback addresses (e.g., 127.0.0.1, ::1)
Link-local and reserved addresses
Non-HTTP/HTTPS schemes (e.g., file://, ftp://, javascript:)
Developed with the Model Context Protocol.