WebSurfer MCP

🌐 WebSurfer MCP

A powerful Model Context Protocol (MCP) server that enables Large Language Models (LLMs) to fetch and extract readable text content from web pages. This tool provides a secure, efficient, and feature-rich way for AI assistants to access web content through a standardized interface.

✨ Features

🔒 Secure URL Validation: Blocks dangerous schemes, private IPs, and localhost domains
📄 Smart Content Extraction: Extracts clean, readable text from HTML pages using advanced parsing
⚡ Rate Limiting: Built-in rate limiting to prevent abuse (60 requests/minute)
🛡️ Content Type Filtering: Only processes supported content types (HTML, plain text, XML)
📏 Size Limits: Configurable content size limits (default: 10MB)
⏱️ Timeout Management: Configurable request timeouts with validation
🔧 Comprehensive Error Handling: Detailed error messages for various failure scenarios
🧪 Full Test Coverage: 45+ unit tests covering all functionality

🏗️ Architecture

The project consists of several key components:

Core Components

MCPURLSearchServer: Main MCP server implementation
TextExtractor: Handles web content fetching and text extraction
URLValidator: Validates and sanitizes URLs for security
Config: Centralized configuration management

Key Features

Async/Await: Built with modern Python async patterns for high performance
Resource Management: Proper cleanup of network connections and resources
Context Managers: Safe resource handling with automatic cleanup
Logging: Comprehensive logging for debugging and monitoring

🚀 Installation

Prerequisites

Python 3.12 or higher
uv package manager (recommended)

Quick Start

Clone the repository:
git clone https://github.com/crybo-rybo/websurfer-mcp cd websurfer-mcp
Install dependencies:
uv sync
Verify installation:
uv run python -c "import mcp_url_search_server; print('Installation successful!')"

🎯 Usage

Starting the MCP Server

The server communicates via stdio (standard input/output) and can be integrated with any MCP-compatible client.

# Start the server
uv run run_server.py serve

# Start with custom log level
uv run run_server.py serve --log-level DEBUG

Testing URL Search Functionality

Test the URL search functionality directly:

# Test with a simple URL
uv run run_server.py test --url "https://example.com"

# Test with custom timeout
uv run run_server.py test --url "https://httpbin.org/html" --timeout 15

Example Test Output

{
  "success": true,
  "url": "https://example.com",
  "title": "Example Domain",
  "content_type": "text/html",
  "status_code": 200,
  "text_length": 1250,
  "text_preview": "Example Domain This domain is for use in illustrative examples in documents..."
}

🛠️ Configuration

The server can be configured using environment variables:

Variable	Default	Description
`MCP_DEFAULT_TIMEOUT`	`10`	Default request timeout in seconds
`MCP_MAX_TIMEOUT`	`60`	Maximum allowed timeout in seconds
`MCP_USER_AGENT`	`MCP-URL-Search-Server/1.0.0`	User agent string for requests
`MCP_MAX_CONTENT_LENGTH`	`10485760`	Maximum content size in bytes (10MB)

Example Configuration

export MCP_DEFAULT_TIMEOUT=15
export MCP_MAX_CONTENT_LENGTH=5242880  # 5MB
uv run run_server.py serve

🧪 Testing

Running All Tests

# Run all tests with verbose output
uv run python -m unittest discover tests -v

# Run tests with coverage (if coverage is installed)
uv run coverage run -m unittest discover tests
uv run coverage report

Running Specific Test Files

# Run only integration tests
uv run python -m unittest tests.test_integration -v

# Run only text extraction tests
uv run python -m unittest tests.test_text_extractor -v

# Run only URL validation tests
uv run python -m unittest tests.test_url_validator -v

Test Results

All 45 tests should pass successfully:

test_content_types_immutable (test_config.TestConfig.test_content_types_immutable) ... ok
test_default_configuration_values (test_config.TestConfig.test_default_configuration_values) ... ok
test_404_error_handling (test_integration.TestMCPURLSearchIntegration.test_404_error_handling) ... ok
...
----------------------------------------------------------------------
Ran 45 tests in 1.827s

OK

🔧 Development

Project Structure

websurfer-mcp/
├── mcp_url_search_server.py  # Main MCP server implementation
├── text_extractor.py         # Web content extraction logic
├── url_validator.py          # URL validation and security
├── config.py                 # Configuration management
├── run_server.py             # Command-line interface
├── run_tests.py              # Test runner utilities
├── tests/                    # Test suite
│   ├── test_integration.py   # Integration tests
│   ├── test_text_extractor.py # Text extraction tests
│   ├── test_url_validator.py # URL validation tests
│   └── test_config.py        # Configuration tests
├── pyproject.toml            # Project configuration
└── README.md                 # This file

🔒 Security Features

URL Validation

Scheme Blocking: Blocks file://, javascript:, ftp:// schemes
Private IP Protection: Blocks access to private IP ranges (10.x.x.x, 192.168.x.x, etc.)
Localhost Protection: Blocks localhost and local domain access
URL Length Limits: Prevents extremely long URLs
Format Validation: Ensures proper URL structure

Content Safety

Content Type Filtering: Only processes supported text-based content types
Size Limits: Configurable maximum content size (default: 10MB)
Rate Limiting: Prevents abuse with configurable limits
Timeout Protection: Configurable request timeouts

📊 Performance

Async Processing: Non-blocking I/O for high concurrency
Connection Pooling: Efficient HTTP connection reuse
DNS Caching: Reduces DNS lookup overhead
Resource Cleanup: Automatic cleanup prevents memory leaks

🙏 Acknowledgments

Built with the Model Context Protocol (MCP)
Uses aiohttp for async HTTP requests
Leverages trafilatura for content extraction
Powered by BeautifulSoup for HTML parsing

Happy web surfing with your AI assistant! 🚀

This server cannot be installed

security - not tested

license - not found

quality - not tested

How are these scores calculated?

remote-capable server

The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.

A Model Context Protocol server that enables AI assistants to securely fetch and extract readable text content from web pages through a standardized interface.

Related MCP Servers

Playwright MCP Server
Kotelberg
-
security
F
license
-
quality
A server that enables AI systems to browse, retrieve content from, and interact with web pages through the Model Context Protocol.
Last updated -
MCP Toolkit
zxfgds
-
security
F
license
-
quality
A comprehensive Model Context Protocol server implementation that enables AI assistants to interact with file systems, databases, GitHub repositories, web resources, and system tools while maintaining security and control.
Last updated -
16
TypeScript
WebSearch-MCP
mnhlt
A
security
F
license
A
quality
A Model Context Protocol server that enables AI assistants to perform real-time web searches, retrieving up-to-date information from the internet via a Crawler API.
Last updated -
1
44
8
JavaScript
Prysm MCP Server
pinkpixel-dev
A
security
A
license
A
quality
A Model Context Protocol server enabling AI assistants to scrape web content with high accuracy and flexibility, supporting multiple scraping modes and content formatting options.
Last updated -
4
17
2
TypeScript
MIT License

View all related MCP servers