Used for making async HTTP requests to web pages, enabling efficient fetching of web content
Blocks javascript: URL schemes as part of the security features to prevent potential security vulnerabilities
Built with modern Python async patterns for high performance, requiring Python 3.12 or higher to run
Supports processing XML content types, allowing extraction of text from XML-based web pages
🌐 WebSurfer MCP
A powerful Model Context Protocol (MCP) server that enables Large Language Models (LLMs) to fetch and extract readable text content from web pages. This tool provides a secure, efficient, and feature-rich way for AI assistants to access web content through a standardized interface.
✨ Features
- 🔒 Secure URL Validation: Blocks dangerous schemes, private IPs, and localhost domains
- 📄 Smart Content Extraction: Extracts clean, readable text from HTML pages using advanced parsing
- ⚡ Rate Limiting: Built-in rate limiting to prevent abuse (60 requests/minute)
- 🛡️ Content Type Filtering: Only processes supported content types (HTML, plain text, XML)
- 📏 Size Limits: Configurable content size limits (default: 10MB)
- ⏱️ Timeout Management: Configurable request timeouts with validation
- 🔧 Comprehensive Error Handling: Detailed error messages for various failure scenarios
- 🧪 Full Test Coverage: 45+ unit tests covering all functionality
🏗️ Architecture
The project consists of several key components:
Core Components
MCPURLSearchServer
: Main MCP server implementationTextExtractor
: Handles web content fetching and text extractionURLValidator
: Validates and sanitizes URLs for securityConfig
: Centralized configuration management
Key Features
- Async/Await: Built with modern Python async patterns for high performance
- Resource Management: Proper cleanup of network connections and resources
- Context Managers: Safe resource handling with automatic cleanup
- Logging: Comprehensive logging for debugging and monitoring
🚀 Installation
Prerequisites
- Python 3.12 or higher
- uv package manager (recommended)
Quick Start
- Clone the repository:
- Install dependencies:
- Verify installation:
🎯 Usage
Starting the MCP Server
The server communicates via stdio (standard input/output) and can be integrated with any MCP-compatible client.
Testing URL Search Functionality
Test the URL search functionality directly:
Example Test Output
🛠️ Configuration
The server can be configured using environment variables:
Variable | Default | Description |
---|---|---|
MCP_DEFAULT_TIMEOUT | 10 | Default request timeout in seconds |
MCP_MAX_TIMEOUT | 60 | Maximum allowed timeout in seconds |
MCP_USER_AGENT | MCP-URL-Search-Server/1.0.0 | User agent string for requests |
MCP_MAX_CONTENT_LENGTH | 10485760 | Maximum content size in bytes (10MB) |
Example Configuration
🧪 Testing
Running All Tests
Running Specific Test Files
Test Results
All 45 tests should pass successfully:
🔧 Development
Project Structure
🔒 Security Features
URL Validation
- Scheme Blocking: Blocks
file://
,javascript:
,ftp://
schemes - Private IP Protection: Blocks access to private IP ranges (10.x.x.x, 192.168.x.x, etc.)
- Localhost Protection: Blocks localhost and local domain access
- URL Length Limits: Prevents extremely long URLs
- Format Validation: Ensures proper URL structure
Content Safety
- Content Type Filtering: Only processes supported text-based content types
- Size Limits: Configurable maximum content size (default: 10MB)
- Rate Limiting: Prevents abuse with configurable limits
- Timeout Protection: Configurable request timeouts
📊 Performance
- Async Processing: Non-blocking I/O for high concurrency
- Connection Pooling: Efficient HTTP connection reuse
- DNS Caching: Reduces DNS lookup overhead
- Resource Cleanup: Automatic cleanup prevents memory leaks
🙏 Acknowledgments
- Built with the Model Context Protocol (MCP)
- Uses aiohttp for async HTTP requests
- Leverages trafilatura for content extraction
- Powered by BeautifulSoup for HTML parsing
Happy web surfing with your AI assistant! 🚀
This server cannot be installed
remote-capable server
The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.
A Model Context Protocol server that enables AI assistants to securely fetch and extract readable text content from web pages through a standardized interface.
Related MCP Servers
- -securityFlicense-qualityA server that enables AI systems to browse, retrieve content from, and interact with web pages through the Model Context Protocol.Last updated -
- -securityFlicense-qualityA comprehensive Model Context Protocol server implementation that enables AI assistants to interact with file systems, databases, GitHub repositories, web resources, and system tools while maintaining security and control.Last updated -16TypeScript
- AsecurityFlicenseAqualityA Model Context Protocol server that enables AI assistants to perform real-time web searches, retrieving up-to-date information from the internet via a Crawler API.Last updated -1448JavaScript
- AsecurityAlicenseAqualityA Model Context Protocol server enabling AI assistants to scrape web content with high accuracy and flexibility, supporting multiple scraping modes and content formatting options.Last updated -4172TypeScriptMIT License