-
securityF
license-
qualityA server that enables AI systems to browse, retrieve content from, and interact with web pages through the Model Context Protocol.
Last updated -
Used for making async HTTP requests to web pages, enabling efficient fetching of web content
Blocks javascript: URL schemes as part of the security features to prevent potential security vulnerabilities
Built with modern Python async patterns for high performance, requiring Python 3.12 or higher to run
Supports processing XML content types, allowing extraction of text from XML-based web pages
π WebSurfer MCP
A powerful Model Context Protocol (MCP) server that enables Large Language Models (LLMs) to fetch and extract readable text content from web pages. This tool provides a secure, efficient, and feature-rich way for AI assistants to access web content through a standardized interface.
β¨ Features
π Secure URL Validation: Blocks dangerous schemes, private IPs, and localhost domains
π Smart Content Extraction: Extracts clean, readable text from HTML pages using advanced parsing
β‘ Rate Limiting: Built-in rate limiting to prevent abuse (60 requests/minute)
π‘οΈ Content Type Filtering: Only processes supported content types (HTML, plain text, XML)
π Size Limits: Configurable content size limits (default: 10MB)
β±οΈ Timeout Management: Configurable request timeouts with validation
π§ Comprehensive Error Handling: Detailed error messages for various failure scenarios
π§ͺ Full Test Coverage: 45+ unit tests covering all functionality
ποΈ Architecture
The project consists of several key components:
Core Components
MCPURLSearchServer: Main MCP server implementationTextExtractor: Handles web content fetching and text extractionURLValidator: Validates and sanitizes URLs for securityConfig: Centralized configuration management
Key Features
Async/Await: Built with modern Python async patterns for high performance
Resource Management: Proper cleanup of network connections and resources
Context Managers: Safe resource handling with automatic cleanup
Logging: Comprehensive logging for debugging and monitoring
π Installation
Prerequisites
Python 3.12 or higher
uv package manager (recommended)
Quick Start
Clone the repository:
git clone https://github.com/crybo-rybo/websurfer-mcp cd websurfer-mcpInstall dependencies:
uv syncVerify installation:
uv run python -c "import mcp_url_search_server; print('Installation successful!')"
π― Usage
Starting the MCP Server
The server communicates via stdio (standard input/output) and can be integrated with any MCP-compatible client.
# Start the server
uv run run_server.py serve
# Start with custom log level
uv run run_server.py serve --log-level DEBUG
Testing URL Search Functionality
Test the URL search functionality directly:
# Test with a simple URL
uv run run_server.py test --url "https://example.com"
# Test with custom timeout
uv run run_server.py test --url "https://httpbin.org/html" --timeout 15
Example Test Output
{
"success": true,
"url": "https://example.com",
"title": "Example Domain",
"content_type": "text/html",
"status_code": 200,
"text_length": 1250,
"text_preview": "Example Domain This domain is for use in illustrative examples in documents..."
}
π οΈ Configuration
The server can be configured using environment variables:
Variable | Default | Description |
|
| Default request timeout in seconds |
|
| Maximum allowed timeout in seconds |
|
| User agent string for requests |
|
| Maximum content size in bytes (10MB) |
Example Configuration
export MCP_DEFAULT_TIMEOUT=15
export MCP_MAX_CONTENT_LENGTH=5242880 # 5MB
uv run run_server.py serve
π§ͺ Testing
Running All Tests
# Run all tests with verbose output
uv run python -m unittest discover tests -v
# Run tests with coverage (if coverage is installed)
uv run coverage run -m unittest discover tests
uv run coverage report
Running Specific Test Files
# Run only integration tests
uv run python -m unittest tests.test_integration -v
# Run only text extraction tests
uv run python -m unittest tests.test_text_extractor -v
# Run only URL validation tests
uv run python -m unittest tests.test_url_validator -v
Test Results
All 45 tests should pass successfully:
test_content_types_immutable (test_config.TestConfig.test_content_types_immutable) ... ok
test_default_configuration_values (test_config.TestConfig.test_default_configuration_values) ... ok
test_404_error_handling (test_integration.TestMCPURLSearchIntegration.test_404_error_handling) ... ok
...
----------------------------------------------------------------------
Ran 45 tests in 1.827s
OK
π§ Development
Project Structure
websurfer-mcp/
βββ mcp_url_search_server.py # Main MCP server implementation
βββ text_extractor.py # Web content extraction logic
βββ url_validator.py # URL validation and security
βββ config.py # Configuration management
βββ run_server.py # Command-line interface
βββ run_tests.py # Test runner utilities
βββ tests/ # Test suite
β βββ test_integration.py # Integration tests
β βββ test_text_extractor.py # Text extraction tests
β βββ test_url_validator.py # URL validation tests
β βββ test_config.py # Configuration tests
βββ pyproject.toml # Project configuration
βββ README.md # This file
π Security Features
URL Validation
Scheme Blocking: Blocks
file://,javascript:,ftp://schemesPrivate IP Protection: Blocks access to private IP ranges (10.x.x.x, 192.168.x.x, etc.)
Localhost Protection: Blocks localhost and local domain access
URL Length Limits: Prevents extremely long URLs
Format Validation: Ensures proper URL structure
Content Safety
Content Type Filtering: Only processes supported text-based content types
Size Limits: Configurable maximum content size (default: 10MB)
Rate Limiting: Prevents abuse with configurable limits
Timeout Protection: Configurable request timeouts
π Performance
Async Processing: Non-blocking I/O for high concurrency
Connection Pooling: Efficient HTTP connection reuse
DNS Caching: Reduces DNS lookup overhead
Resource Cleanup: Automatic cleanup prevents memory leaks
π Acknowledgments
Built with the Model Context Protocol (MCP)
Uses aiohttp for async HTTP requests
Leverages trafilatura for content extraction
Powered by BeautifulSoup for HTML parsing
Happy web surfing with your AI assistant! π
remote-capable server
The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.
Tools
A Model Context Protocol server that enables AI assistants to securely fetch and extract readable text content from web pages through a standardized interface.
Related MCP Servers
- -securityFlicense-qualityA comprehensive Model Context Protocol server implementation that enables AI assistants to interact with file systems, databases, GitHub repositories, web resources, and system tools while maintaining security and control.Last updated -361
- AsecurityFlicenseAqualityA Model Context Protocol server that enables AI assistants to perform real-time web searches, retrieving up-to-date information from the internet via a Crawler API.Last updated -123517
- AsecurityAlicenseAqualityA Model Context Protocol server enabling AI assistants to scrape web content with high accuracy and flexibility, supporting multiple scraping modes and content formatting options.Last updated -4292MIT License