# MCP HTTP Fetcher Server
A Model Context Protocol (MCP) server that fetches web pages and converts them to Markdown format. This server supports both **SSE (Server-Sent Events)** and **Stdio** protocols, making it suitable for web deployments, Kubernetes environments, desktop clients, and sidecar patterns.
## Features
- ✅ **Dual Protocol Support**: SSE (default) and Stdio protocols
- ✅ **HTTP/HTTPS URL fetching**: Download content from any web URL
- ✅ **HTML to Markdown conversion**: Clean conversion using html2text
- ✅ **MCP Protocol compliance**: Follows MCP server standards
- ✅ **Kubernetes ready**: SSE protocol with health checks and graceful scaling
- ✅ **Sidecar friendly**: Stdio protocol for process-to-process communication
- ✅ **Error handling**: Robust error handling for network issues
- ✅ **Docker support**: Ready-to-run Docker container
- ✅ **Async operation**: Non-blocking HTTP requests
- ✅ **Modular architecture**: Clean separation of concerns for maintainability
## Quick Start
### SSE Protocol (Default - Web/Kubernetes Deployments)
```bash
# Start with SSE protocol (default)
python app/server.py
# Custom host/port for Kubernetes
python app/server.py --protocol sse --host 0.0.0.0 --port 8000
```
### Stdio Protocol (Desktop Clients/Sidecars)
```bash
# Start with stdio protocol
python app/server.py --protocol stdio
```
## Protocol Overview
| Protocol | Use Case | Deployment | Scaling |
|----------|----------|------------|---------|
| **SSE (Default)** | Web apps, APIs, Kubernetes | HTTP server with endpoints | Horizontal scaling, load balancing |
| **Stdio** | Desktop clients, sidecars | Process communication | Single client per process |
For detailed protocol documentation, see [docs/PROTOCOLS.md](docs/PROTOCOLS.md).
## Project Structure
The project follows Python best practices with a clean, modular architecture:
```
mcp_fetcher_http/
├── app/ # Main application package
│ ├── core/ # Core business logic
│ │ ├── fetcher.py # URL fetching functionality
│ │ └── converter.py # HTML to Markdown conversion
│ ├── protocols/ # Protocol implementations
│ │ ├── base.py # Abstract protocol interface
│ │ ├── stdio.py # Standard I/O protocol implementation
│ │ └── sse.py # Server-Sent Events protocol implementation
│ └── server.py # New modular server entry point
├── docs/ # Documentation
│ └── PROTOCOLS.md # Comprehensive protocol guide
├── tests/ # Test suite
│ ├── test_fetcher.py # Tests for URL fetching
│ ├── test_converter.py # Tests for HTML conversion
│ └── test_server.py # Integration tests
├── examples/ # Example usage and demos
│ ├── demo.py # Functionality demonstration
│ └── mcp_config_example.json
├── server.py # Backward-compatible entry point
├── requirements.txt # Python dependencies
└── Dockerfile # Container configuration
```
### Entry Points
- **`python app/server.py`** - Primary entry point with full protocol support
- **`python server.py`** - Legacy entry point (limited protocol options)
- **`./run_server.sh`** - Convenience script with dependency management
## Installation
### Option 1: Direct Python Installation
1. Clone the repository:
```bash
git clone https://github.com/photonn/mcp_fetcher_http.git
cd mcp_fetcher_http
```
2. Install dependencies:
```bash
pip install -r requirements.txt
```
3. Run the server:
```bash
# SSE protocol (default - for web/Kubernetes deployments)
python app/server.py
# Stdio protocol (for desktop clients)
python app/server.py --protocol stdio
# Custom SSE configuration
python app/server.py --protocol sse --host 0.0.0.0 --port 8080
# OR use the convenience script:
./run_server.sh
```
### Option 2: Docker Installation
1. Build the Docker image:
```bash
docker build -t mcp-fetcher-http .
```
2. Run the container:
```bash
# SSE protocol (default)
docker run -p 8000:8000 mcp-fetcher-http
# Stdio protocol
docker run -it mcp-fetcher-http python app/server.py --protocol stdio
```
**Note:** Docker build may require internet access to install Python packages. In restricted environments, use the direct Python installation method.
## Usage
The server exposes a single MCP tool called `fetch_url` that accepts a URL parameter and returns the page content converted to Markdown.
### MCP Client Configuration
#### For SSE Protocol (Web/Kubernetes deployments)
```json
{
"mcpServers": {
"http-fetcher": {
"command": "curl",
"args": ["-N", "-H", "Accept: text/event-stream", "http://localhost:8000/sse"],
"description": "HTTP fetcher using SSE protocol"
}
}
}
```
#### For Stdio Protocol (Desktop clients)
```json
{
"mcpServers": {
"http-fetcher": {
"command": "python",
"args": ["/path/to/mcp_fetcher_http/app/server.py", "--protocol", "stdio"],
"description": "HTTP fetcher using stdio protocol"
}
}
}
```
### Command Line Options
```bash
python app/server.py --help
```
Available options:
- `--protocol {stdio,sse}` - Communication protocol (default: sse)
- `--host HOST` - Host to bind SSE server (default: localhost)
- `--port PORT` - Port for SSE server (default: 8000)
- `--endpoint ENDPOINT` - SSE message endpoint (default: /messages)
- `--server-name NAME` - Server identifier
- `--server-version VERSION` - Server version
### Tool Schema
```json
{
"name": "fetch_url",
"description": "Fetch a web page and convert it to Markdown format",
"inputSchema": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "The URL of the web page to fetch and convert to Markdown"
}
},
"required": ["url"]
}
}
```
### Example Usage
When connected to an MCP client, you can use the tool like this:
```json
{
"name": "fetch_url",
"arguments": {
"url": "https://example.com"
}
}
```
The server will return the page content converted to Markdown format.
## Testing
Run the included test scripts to verify functionality:
```bash
# Run all tests (recommended)
python tests/test_server.py # Integration tests
python tests/test_fetcher.py # URL fetching tests
python tests/test_converter.py # HTML conversion tests
# Run demo to see HTML to Markdown conversion
python examples/demo.py
```
### Testing the MCP Server
The server communicates via stdin/stdout using the MCP protocol. For basic testing:
1. Start the server: `python server.py` (or `python app/server.py`)
2. The server will wait for MCP protocol messages on stdin
3. Send a list_tools request to see available tools
4. Send a call_tool request with the fetch_url tool
For easier testing, use an MCP client like Claude Desktop or develop a simple test client.
## Configuration
The server includes the following configurable aspects:
- **Timeout**: 30-second timeout for HTTP requests
- **Content types**: Accepts HTML and XML content types
- **Markdown conversion**: Preserves links and images in output
## Error Handling
The server handles various error conditions:
- Invalid URLs
- Network connectivity issues
- HTTP error responses (4xx, 5xx)
- Content parsing errors
- Timeout conditions
## Security Considerations
- The server validates URLs before processing
- Runs in a non-root Docker container
- Has reasonable timeout limits to prevent hanging
- Does not execute or process JavaScript content
## Dependencies
- `mcp>=1.0.0` - Model Context Protocol server framework
- `aiohttp>=3.9.0` - Async HTTP client for fetching URLs
- `html2text>=2020.1.16` - HTML to Markdown conversion
- `typing-extensions>=4.0.0` - Type hints support
## License
Licensed under the Apache License, Version 2.0. See the LICENSE file for details.
