# Hermes SearXNG MCP Server

> Clean, production-ready MCP server for self-hosted SearXNG
> Designed for Hermes Agent, compatible with Claude Code, OpenClaw, and other MCP clients

[![Python Version](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
[![MCP](https://img.shields.io/badge/MCP-1.0%2B-purple)](https://modelcontextprotocol.io/)

## Overview

This MCP server provides web search and content retrieval capabilities using a self-hosted [SearXNG](https://searxng.org/) instance. It's designed to be a drop-in replacement for paid web search APIs, offering:

- **Privacy-preserving**: Self-hosted, no data sent to third parties
- **Cost-free**: No API keys, no rate limits
- **Flexible**: 70+ search engines aggregated through SearXNG
- **MCP-compliant**: Works with any MCP client (Hermes, Claude Desktop, OpenClaw, etc.)
- **Content extraction**: Automatic Markdown extraction from search results

## Features

### Web Search (`web_search`)
- Search across 70+ search engines via SearXNG
- Support for multiple categories (general, news, images, videos, IT, science, etc.)
- Language filtering
- Time-based filtering (day, week, month, year)
- Optional full content fetching for results

### Content Retrieval (`get_content`)
- Extract LLM-ready Markdown from any URL
- Specialized handlers for:
  - GitHub Issues
  - Stack Exchange (Stack Overflow, Server Fault, Super User, etc.)
  - Wikipedia
  - arXiv
- Universal fallback using Trafilatura + BeautifulSoup
- Handles paywalls and anti-bot measures (best-effort)

### News Search (`search_news`)
- Convenience wrapper optimized for news articles
- Default time filtering (last week)
- Aggregates from news-focused search engines

### Security & Production Features
- **SSRF Protection**: Blocks private IPs, localhost, cloud metadata endpoints (169.254.169.254)
- **Input Validation**: All tool parameters validated (query length, categories, language codes, time ranges)
- **Response Size Limits**: Configurable max content size (default: 5MB) with streaming
- **Connection Pooling**: Shared HTTP client with keep-alive connections and graceful shutdown
- **In-Memory Caching**: 5-minute TTL cache for search results (100 entries max)
- **Docker Support**: Multi-stage Dockerfile with docker-compose (includes SearXNG sidecar)
- **CI/CD**: GitHub Actions workflow (Python 3.10/3.11/3.12, lint, typecheck, test)

## Installation

### From Source

```bash
# Clone the repository
git clone https://github.com/EmericLaberge/hermes-searxng-mcp.git
cd hermes-searxng-mcp

# Install in development mode
pip install -e .
```

### From PyPI (when published)

```bash
pip install hermes-searxng-mcp-server
```

## Configuration

### SearXNG Instance

You need a running SearXNG instance. If you don't have one:

```bash
# Using Docker
docker run -d -p 8888:8080 \
  -v $(pwd)/searxng:/etc/searxng \
  searxng/searxng:latest

# Or follow the official guide:
# https://docs.searxng.org/admin/installation.html
```

Make sure JSON output is enabled in your SearXNG `settings.yml`:

```yaml
search:
  formats:
    - html
    - json  # Required!
```

### Environment Variables

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `SEARXNG_BASE_URL` | Yes | - | URL of your SearXNG instance (e.g., `http://localhost:8888`) |
| `SEARXNG_TIMEOUT_SECONDS` | No | 30 | Request timeout in seconds |
| `SEARXNG_USER_AGENT` | No | Chrome UA | Custom User-Agent header |
| `SEARXNG_HEADERS_JSON` | No | {} | JSON object with custom HTTP headers |
| `SEARXNG_MAX_CONTENT_BYTES` | No | 5242880 | Maximum content size in bytes (default: 5MB) |
| `SEARXNG_TRANSPORT` | No | stdio | Transport type: `stdio` or `http` |
| `SEARXNG_HOST` | No | 0.0.0.0 | HTTP host (only for HTTP transport) |
| `SEARXNG_PORT` | No | 8000 | HTTP port (only for HTTP transport) |

### Proxy Support

The server supports proxy configuration through standard environment variables. Proxy handling is implemented natively by [httpx](https://www.python-httpx.org/) via its `trust_env` feature (enabled by default).

**Standard proxy variables:**

| Variable | Description |
|----------|-------------|
| `HTTP_PROXY` | Proxy URL for HTTP requests (e.g., `http://proxy:8080`) |
| `HTTPS_PROXY` | Proxy URL for HTTPS requests (e.g., `http://proxy:8080`) |
| `NO_PROXY` | Comma-separated list of hosts to bypass the proxy |

**NO_PROXY patterns supported:**
- Exact host match: `localhost`, `searxng.local`
- Domain suffix: `.example.com` (matches `example.com`, `www.example.com`)
- Wildcard: `*` (bypasses all hosts)
- Port-based: `localhost:8888`

**Proxy authentication:**

Include credentials in the proxy URL:
```bash
export HTTP_PROXY="http://user:pass@proxy:8080"
export HTTPS_PROXY="http://user:pass@proxy:8080"
```

**TLS certificates:**

For corporate proxies that inspect SSL traffic, configure custom CA certificates:
```bash
export SSL_CERT_FILE=/path/to/ca-bundle.crt
# or
export REQUESTS_CA_BUNDLE=/path/to/ca-bundle.crt
```

Note: The proxy is used for outbound connections from the MCP server to SearXNG and fetched URLs. The SearXNG instance itself should be configured separately if it also needs proxy access.

## Usage

### With Hermes Agent

```bash
# Add the MCP server
hermes mcp add searxng \
  --command python3 \
  --args -m hermes_searxng_mcp.server

# Restart Hermes to load the server
hermes gateway restart
```

Then in Hermes, just ask naturally:
```
You: Search for the latest news about AI
Hermes: [Uses search_news tool automatically]

You: Find documentation on FastAPI
Hermes: [Uses web_search tool automatically]

You: What's the content of https://example.com?
Hermes: [Uses get_content tool automatically]
```

### With Claude Desktop

Add to `~/.config/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "searxng": {
      "command": "python3",
      "args": ["-m", "hermes_searxng_mcp.server"],
      "env": {
        "SEARXNG_BASE_URL": "http://localhost:8888"
      }
    }
  }
}
```

Restart Claude Desktop.

### With OpenClaw

Add to your OpenClaw MCP configuration:

```yaml
mcp_servers:
  searxng:
    command: python3
    args: ["-m", "hermes_searxng_mcp.server"]
    env:
      SEARXNG_BASE_URL: "http://localhost:8888"
```

### Standalone Testing

```bash
# Set environment variable
export SEARXNG_BASE_URL="http://localhost:8888"

# Run the server
python -m hermes_searxng_mcp.server

# In another terminal, test with MCP client
# or use curl for manual testing of SearXNG
curl "http://localhost:8888/search?q=test&format=json"
```

### With Docker

```bash
# Build and run with SearXNG sidecar
docker compose up -d

# Or build manually
docker build -t hermes-searxng-mcp .
docker run -e SEARXNG_BASE_URL=http://searxng:8080 hermes-searxng-mcp
```

### HTTP Transport

For remote access, set `SEARXNG_TRANSPORT=http`:

```bash
export SEARXNG_TRANSPORT=http
export SEARXNG_HOST=0.0.0.0
export SEARXNG_PORT=8000
python -m hermes_searxng_mcp.server
```

Test with curl:
```bash
curl http://localhost:8000/mcp
```

## Tools Reference

### `web_search`

Search the web and return results.

**Parameters:**
- `query` (string, required): Search query
- `num_results` (int, optional): Number of results (default: 5, max: 50)
- `categories` (string, optional): Category - `general`, `news`, `images`, `videos`, `it`, `science`, `files`, `music`
- `language` (string, optional): Language code (default: `"en"`)
- `time_range` (string, optional): Time filter - `day`, `week`, `month`, `year`
- `include_content` (bool, optional): Fetch full page content (default: `false`)

**Returns:**
```json
{
  "results": [
    {
      "title": "Result Title",
      "url": "https://example.com",
      "snippet": "Search snippet from SearXNG...",
      "content": "# Full Markdown content...",
      "engine": ["google", "bing"]
    }
  ]
}
```

### `get_content`

Fetch a URL and extract Markdown content.

**Parameters:**
- `url` (string, required): URL to fetch

**Returns:**
```json
{
  "url": "https://example.com",
  "content": "# Extracted Markdown content..."
}
```

### `search_news`

Search recent news articles (convenience wrapper).

**Parameters:**
- `query` (string, required): News topic
- `num_results` (int, optional): Number of articles (default: 10)
- `time_range` (string, optional): Time filter (default: `"week"`)
- `language` (string, optional): Language code (default: `"en"`)

**Returns:** Same as `web_search`

## Development

### Setup

```bash
# Clone repository
git clone https://github.com/EmericLaberge/hermes-searxng-mcp.git
cd hermes-searxng-mcp

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install in development mode with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check .
ruff format .
```

### Project Structure

```
hermes-searxng-mcp/
├── src/hermes_searxng_mcp/
│   ├── __init__.py          # Package init
│   ├── server.py            # MCP server, tools, and input validation
│   ├── searxng.py           # SearXNG search with caching
│   ├── content_extractor.py # Content extraction with size limits
│   ├── http_client.py       # Shared HTTP client (connection pooling)
│   ├── ssrf_protection.py   # SSRF URL validation
│   ├── cache.py             # In-memory TTL cache
│   └── models.py            # Pydantic models
├── tests/                   # Test suite (41 tests)
├── .github/workflows/ci.yml # CI/CD pipeline
├── Dockerfile               # Multi-stage Docker build
├── docker-compose.yml       # Docker Compose with SearXNG
├── pyproject.toml           # Project configuration
├── README.md                # This file
└── LICENSE                  # MIT license
```

### Adding New Features

1. **Add specialized content handler**: Extend `content_extractor.py` with new patterns
2. **Add new search categories**: Update `searxng.py` to support more SearXNG categories
3. **Add new MCP tools**: Add `@mcp.tool()` decorators in `server.py`

## Troubleshooting

### "SEARXNG_BASE_URL is not set"

Set the environment variable:
```bash
export SEARXNG_BASE_URL="http://localhost:8888"
```

### "SearXNG returned 403 Forbidden"

JSON output is disabled in SearXNG. Enable it in `settings.yml`:
```yaml
search:
  formats:
    - html
    - json  # Add this line
```

Then restart SearXNG.

### "Could not fetch content"

This is normal for:
- Paywalled content (NYT, WSJ, etc.)
- Protected content (requires login)
- Unsupported content types (PDFs, videos)
- Bot detection (some sites block automated requests)

### Tools not appearing in Hermes

```bash
# Check MCP server is running
hermes mcp list

# Test connection
hermes mcp test searxng

# Restart Hermes
hermes gateway restart
```

### Slow performance

- Reduce `num_results` parameter
- Set `include_content=False` (default)
- Check SearXNG instance performance
- Reduce `SEARXNG_TIMEOUT_SECONDS` if network is slow

## Comparison with Alternatives

| Feature | Hermes SearXNG MCP | Serper/Tavily | Built-in Web Search |
|---------|-------------------|---------------|-------------------|
| **Self-hosted** | ✅ Yes | ❌ No | ❌ No |
| **Privacy** | ✅ No data leaves | ❌ Sends to third-party | ❌ Sends to third-party |
| **Cost** | ✅ Free | ❌ Paid | ❌ Paid (often) |
| **Rate Limits** | ✅ None | ⚠️ Limited | ⚠️ Limited |
| **Search Engines** | ✅ 70+ via SearXNG | ⚠️ Google only | ⚠️ Varies |
| **Content Extraction** | ✅ Yes | ❌ No | ⚠️ Varies |
| **News Search** | ✅ Yes | ✅ Yes | ⚠️ Varies |
| **MCP Compliant** | ✅ Yes | ✅ Yes | ✅ Yes |
| **Docker Support** | ✅ Yes | ❌ No | ❌ No |
| **SSRF Protection** | ✅ Yes | ❌ No | ❌ No |
| **In-Memory Cache** | ✅ Yes | ❌ No | ❌ No |
| **CI/CD** | ✅ GitHub Actions | ⚠️ Varies | ⚠️ Varies |

## Architecture

The server follows a clean, modular architecture:

```
┌─────────────────────────────────────────────────┐
│              MCP Client (Hermes)                │
└───────────────────┬─────────────────────────────┘
                    │ MCP Protocol (stdio)
┌───────────────────▼─────────────────────────────┐
│          FastMCP Server Layer                   │
│  ┌──────────────────────────────────────────┐   │
│  │  @mcp.tool() Decorators                  │   │
│  │  - web_search()                          │   │
│  │  - get_content()                         │   │
│  │  - search_news()                         │   │
│  └──────────────────────────────────────────┘   │
└───────────────────┬─────────────────────────────┘
                    │
        ┌───────────┼───────────┐
        │           │           │
┌───────▼──────┐ ┌─▼────────┐ ┌▼──────────────┐
│  SearXNG     │ │ Content  │ │   Models      │
│  Search      │ │ Extractor│ │   (Pydantic)  │
│  + Cache     │ │ + SSRF   │ │               │
│  + Validation│ │ + Limits │ │               │
└──────────────┘ └──────────┘ └────────────────┘
        │
        ▼
┌──────────────────────┐
│  SearXNG Instance    │
│  (HTTP + JSON API)   │
└──────────────────────┘
```

## Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

## License

MIT License - see [LICENSE](LICENSE) file for details.

## Acknowledgments

- Inspired by [kindly-web-search-mcp-server](https://github.com/Shelpuk-AI-Technology-Consulting/kindly-web-search-mcp-server)
- Built with [FastMCP](https://github.com/jlowin/fastmcp)
- Powered by [SearXNG](https://searxng.org/)
- Content extraction via [Trafilatura](https://github.com/adbar/trafilatura) and [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)

## Support

- **Issues**: [GitHub Issues](https://github.com/EmericLaberge/hermes-searxng-mcp/issues)
- **Discussions**: [GitHub Discussions](https://github.com/EmericLaberge/hermes-searxng-mcp/discussions)
- **SearXNG Docs**: https://docs.searxng.org/
- **MCP Protocol**: https://modelcontextprotocol.io/

---