Skip to main content
Glama
crybo-rybo

WebSurfer MCP

by crybo-rybo
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

WebSurfer is a Model Context Protocol (MCP) server designed to provide Large Language Models (LLMs) with secure and efficient access to web content.

Core Features

  • Advanced URL Validation: Implements strict security controls using the ipaddress module to block private, loopback, link-local, and reserved destinations before any fetch occurs.

  • Optimized Content Extraction: Utilizes trafilatura and BeautifulSoup4 to extract high-quality, readable text from HTML, effectively removing boilerplate such as navigation, headers, and scripts.

  • Resource Management: Enforces strict content size limits and request timeouts to ensure system stability and performance.

  • Redirect Safety: Validates every redirect hop and refuses redirects to blocked schemes, localhost, private IP literals, or unsafe DNS targets.

  • Rate Limiting: Built-in request throttling to prevent service abuse and manage resource consumption.

  • Robust Error Handling: Provides granular feedback for network issues, HTTP errors, and content parsing failures.

Related MCP server: Playwright MCP Server

Project Layout

websurfer-mcp/
├── src/websurfer_mcp/
│   ├── cli.py
│   ├── config.py
│   ├── extractor.py
│   ├── networking.py
│   ├── server.py
│   └── url_validation.py
├── tests/
├── docs/images/
├── pyproject.toml
└── run_tests.py

Key runtime components:

  • WebSurferServer: MCP transport and tool registration.

  • TextExtractor: asynchronous HTTP fetching and readable-text extraction.

  • SafeResolver: DNS resolution guard that rejects private and reserved IP answers.

  • URLValidator: URL normalization and SSRF-focused validation.

  • Config: environment-driven runtime configuration.

Installation

Prerequisites

  • Python 3.12 or higher

  • uv package manager

Setup

  1. Clone the repository:

    git clone https://github.com/crybo-rybo/websurfer-mcp
cd websurfer-mcp

  2. Install runtime dependencies:

    uv sync

  3. Install development tooling:

    uv sync --group dev

Usage

Server Execution

The server communicates via standard I/O (stdio) and is compatible with any MCP-compliant client.

Use either the console script or the package module:

uv run websurfer-mcp serve
uv run python -m websurfer_mcp serve

Manual Testing

You can verify the extraction functionality directly from the command line:

uv run websurfer-mcp test --url "https://example.com"

Desktop Client Integration

Claude Desktop

To use WebSurfer MCP with Claude Desktop, add the following configuration to your claude_desktop_config.json file.

Path locations:

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

  • Windows: %APPDATA%\Claude\claude_desktop_config.json

Configuration:

Replace /path/to/websurfer-mcp with the absolute path to your cloned repository.

After updating the configuration, restart Claude Desktop to enable the search_url tool.

{
  "mcpServers": {
    "websurfer": {
      "command": "uv",
      "args": [
        "--directory",
        "/path/to/websurfer-mcp",
        "run",
        "python",
        "-m",
        "websurfer_mcp",
        "serve"
      ]
    }
  }
}

Configuration

The server can be configured using the following environment variables:

Variable

Default

Description

MCP_DEFAULT_TIMEOUT

10

Default request timeout in seconds.

MCP_MAX_TIMEOUT

60

Maximum allowed timeout in seconds.

MCP_MAX_REDIRECTS

10

Maximum number of redirect hops to follow.

MCP_USER_AGENT

websurfer-mcp/0.2.0

User-Agent string for outgoing requests.

MCP_MAX_CONTENT_LENGTH

10485760

Maximum content size in bytes (default 10MB).

Development

Run the test suite:

uv run pytest

Run quality checks:

uv run ruff check .
uv run ruff format .

Run a focused module:

uv run python run_tests.py --module test_server

Security

WebSurfer MCP is designed with security as a primary concern. It explicitly blocks:

  • Private IP ranges (e.g., 10.0.0.0/8, 192.168.0.0/16)

  • Loopback addresses (e.g., 127.0.0.1, ::1)

  • Link-local and reserved addresses

  • Non-HTTP/HTTPS schemes (e.g., file://, ftp://, javascript:)

  • Redirect hops that resolve to blocked destinations

  • DNS answers that resolve public-looking hostnames to private or reserved IPs

Developed with the Model Context Protocol.

Available Tools

1 tool
search_urlC

Fetch and return plain-text content from a web page URL. Handles various content types and provides comprehensive error handling.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to fetch content from. Must be a valid HTTP/HTTPS URL.
timeoutNoOptional timeout in seconds (default: 10)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'handles various content types' and 'comprehensive error handling,' but lacks details on permissions, rate limits, network dependencies, or what constitutes 'various content types.' This leaves significant gaps in understanding the tool's behavior and constraints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with two sentences that are front-loaded with the core purpose. It avoids unnecessary details but could be slightly more structured by explicitly separating functional and behavioral aspects. Overall, it is efficient with minimal waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity of a web fetching tool with no annotations and no output schema, the description is incomplete. It lacks details on return values, error types, content type handling specifics, and operational constraints. This makes it insufficient for an AI agent to fully understand the tool's context and usage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage, clearly documenting both parameters. The description adds no additional meaning beyond the schema, such as examples or edge cases for the URL or timeout. However, the baseline is 3 since the schema adequately covers parameter semantics without need for compensation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with specific verbs ('fetch and return') and resource ('plain-text content from a web page URL'). It distinguishes the type of content returned (plain-text) and mentions handling various content types. However, without sibling tools, the differentiation aspect is not applicable, preventing a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives, prerequisites, or specific contexts. It mentions 'comprehensive error handling' but does not specify scenarios or limitations, leaving usage decisions unclear for an AI agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 1 tool update
    • First observedsearch_url
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

B3.1/5.0

Scored across 1 tool

Disambiguation5/5

With only one tool, there is no possibility of ambiguity or overlap between tools, making disambiguation perfect.

Naming Consistency5/5

The single tool name 'search_url' follows a clear verb_noun pattern, and with no other tools, consistency is inherently perfect.

Tool Count2/5

One tool is too few for a server named 'WebSurfer MCP', which suggests broader web navigation or interaction capabilities beyond just fetching content from a URL.

Completeness2/5

The tool surface is severely incomplete for a web surfing domain; it lacks basic operations like navigating links, interacting with forms, or handling dynamic content, leaving significant gaps for agent workflows.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

Appeared in Searches