# MCP Websearch Aggregator

## Overview

`mcp-websearch-aggregator` is a Python microservice that exposes a JSON-RPC `/mcp` endpoint and aggregates search results from DuckDuckGo, Wikidata, and ArXiv into a single markdown response.

This repository follows an enterprise-friendly `src/` layout and contains automated tests, configuration via environment variables, and Prometheus-compatible metrics.

## Table of Contents

- [Quick Start](#quick-start)
- [Prerequisites](#prerequisites)
- [Repository Layout](#repository-layout)
- [Configuration](#configuration)
- [Environment Profiles](#environment-profiles)
- [Development](#development)
- [Testing](#testing)
- [Packaging](#packaging)
- [Running](#running)
- [Deployment](#deployment)
- [API Contract](#api-contract)
- [Troubleshooting](#troubleshooting)

## Quick Start

```bash
git clone <repo-url>
cd mcp-websearch-aggregator
uv venv
source .venv/bin/activate
uv sync
python main.py
```

Open `http://127.0.0.1:3000` and send JSON-RPC requests to `/mcp`.

## Prerequisites

### macOS

This project targets Python 3.14. On macOS, install the required runtime and tooling with Homebrew.

```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
brew install python@3.14
brew install uv
```

Verify your installations:

```bash
python3.14 --version
uv --version
```

### Recommended local environment

```bash
uv venv
source .venv/bin/activate
uv sync
```

## Repository Layout

```text
.
├── README.md
├── pyproject.toml
├── main.py
├── src/
│   └── mcp_websearch_aggregator/
│       ├── config.py
│       ├── logging.py
│       ├── main.py
│       ├── middleware.py
│       ├── metrics.py
│       ├── server.py
│       ├── services/
│       │   ├── arxiv.py
│       │   ├── duckduckgo.py
│       │   └── wikidata.py
│       └── utils/
│           └── formatter.py
└── tests/
    ├── integration/
    └── unit/
```

## Configuration

The service is configured using environment variables. Defaults are provided in `src/mcp_websearch_aggregator/config.py`.

| Variable | Default | Description |
|---|---|---|
| `ENVIRONMENT` | `dev` | Runtime environment identifier (`dev`, `uat`, `prod`) |
| `SERVER_HOST` | `127.0.0.1` | Host interface for the HTTP server |
| `SERVER_PORT` | `3000` | Port for the HTTP server |
| `LOG_LEVEL` | `INFO` | Logging verbosity |
| `REQUEST_TIMEOUT` | `10` | HTTP request timeout in seconds |
| `MAX_RESULTS` | `1` | Maximum results fetched per third-party service |
| `DUCKDUCKGO_API` | `https://api.duckduckgo.com` | DuckDuckGo service endpoint |
| `WIKIDATA_API` | `https://www.wikidata.org/w/api.php` | Wikidata service endpoint |
| `ARXIV_API` | `http://export.arxiv.org/api/query` | ArXiv service endpoint |
| `METRICS_ENABLED` | `true` | Enable Prometheus metrics export |

### Default environment variables

```bash
export ENVIRONMENT=dev
export SERVER_HOST=127.0.0.1
export SERVER_PORT=3000
export LOG_LEVEL=DEBUG
export REQUEST_TIMEOUT=10
export MAX_RESULTS=1
export METRICS_ENABLED=true
```

### Recommended config files

Use a dedicated `.env/` folder to organize environment-specific configuration files. This keeps configs organized and scalable as the project grows.

| Profile | Path (from repo root) | Purpose | Key Settings |
|---|---|---|---|
| DEV | `.env/dev` | Local development | `SERVER_HOST=127.0.0.1`, `LOG_LEVEL=DEBUG`, `REQUEST_TIMEOUT=10` |
| UAT | `.env/uat` | User acceptance testing | `SERVER_HOST=0.0.0.0`, `LOG_LEVEL=INFO`, `REQUEST_TIMEOUT=15` |
| PROD | `.env/prod` | Production deployment | `SERVER_HOST=0.0.0.0`, `LOG_LEVEL=INFO`, `REQUEST_TIMEOUT=20` |

Create the folder structure:

```bash
mkdir -p .env/dev .env/uat .env/prod
```

Each file should define the same variables (shown in the `Configuration` section above). Load a profile before running the app:

```bash
source .env/dev
python main.py
```

For UAT and PROD, keep these profile files secure and out of public source control. Alternatively, inject environment variables through your CI/CD or container runtime instead of using local files.

Because profile files already define environment variables, you do not need to run additional `export` commands after sourcing them. The shell will load the values into the current session automatically.

For UAT and PROD, keep these profile files in a secure deployment pipeline, not in public source control. Alternatively, inject the same variables through your CI/CD or container runtime instead of using local files.

## Environment Profiles

This repository supports explicit environment profiles via the `ENVIRONMENT` variable. Each profile is a preset of environment variables tailored for a specific deployment context.

| Setting | DEV | UAT | PROD |
|---|---|---|---|
| `ENVIRONMENT` | `dev` | `uat` | `prod` |
| `SERVER_HOST` | `127.0.0.1` | `0.0.0.0` | `0.0.0.0` |
| `SERVER_PORT` | `3000` | `3000` | `3000` |
| `LOG_LEVEL` | `DEBUG` | `INFO` | `INFO` |
| `REQUEST_TIMEOUT` | `10` | `15` | `20` |
| `MAX_RESULTS` | `1` | `1` | `1` |
| `METRICS_ENABLED` | `true` | `true` | `true` |

## Development

### Install dependencies

```bash
uv sync
```

This reads `pyproject.toml` and installs all runtime and development dependencies into the active virtual environment.

### Recommended workflow

1. Create a virtual environment
2. Activate the environment
3. Install dependencies
4. Run the application locally
5. Execute tests

### Running the app locally

Use the root launcher:

```bash
python main.py
```

Or run the ASGI app directly with Uvicorn:

```bash
uvicorn mcp_websearch_aggregator.server:app --host 127.0.0.1 --port 3000 --reload
```

### Debugging

- Inspect service logs on startup and request handling
- Use the `/metrics` endpoint for runtime metrics
- Increase logging verbosity with `LOG_LEVEL=DEBUG`

## Testing

Run all tests with Pytest:

```bash
uv run pytest -q
```

Run a specific test file:

```bash
uv run pytest tests/unit/test_duckduckgo.py -q
```

### Code Coverage

Measure code coverage with `pytest-cov`:

```bash
uv run pytest --cov=src/mcp_websearch_aggregator --cov-report=term-missing
```

Generate an HTML coverage report:

```bash
uv run pytest --cov=src/mcp_websearch_aggregator --cov-report=html
```

View the report in your browser:

```bash
open htmlcov/index.html
```

Coverage configuration is defined in `.coveragerc`. The project enforces a minimum **80% code coverage** threshold. Excluded lines are marked with `# pragma: no cover`.

**Coverage metrics:**

- Line coverage: % of lines executed
- Branch coverage: % of conditional branches exercised
- Missing lines: source lines not covered by tests

## Packaging

Build a source distribution and wheel:

```bash
python -m pip install --upgrade build
python -m build
```

Install the package locally for validation:

```bash
python -m pip install dist/mcp_websearch_aggregator-0.1.0-py3-none-any.whl
```

## Running

The service exposes two endpoints:

- `POST /mcp` — JSON-RPC search endpoint
- `GET /metrics` — Prometheus metrics endpoint

Example run command:

```bash
uvicorn mcp_websearch_aggregator.server:app --host 0.0.0.0 --port 3000
```

## Deployment

For production, use an ASGI server with process management and explicit environment configuration.

### Deploy to UAT

```bash
export ENVIRONMENT=uat
export SERVER_HOST=0.0.0.0
export SERVER_PORT=3000
export LOG_LEVEL=INFO
export REQUEST_TIMEOUT=15
export MAX_RESULTS=1
export METRICS_ENABLED=true

gunicorn -k uvicorn.workers.UvicornWorker mcp_websearch_aggregator.server:app --bind 0.0.0.0:3000
```

### Deploy to PROD

```bash
export ENVIRONMENT=prod
export SERVER_HOST=0.0.0.0
export SERVER_PORT=3000
export LOG_LEVEL=INFO
export REQUEST_TIMEOUT=20
export MAX_RESULTS=1
export METRICS_ENABLED=true

gunicorn -k uvicorn.workers.UvicornWorker mcp_websearch_aggregator.server:app --bind 0.0.0.0:3000
```

### Docker / containerized deployment

For container deployment, pass environment variables through your container runtime or orchestration platform.

```bash
docker run -e ENVIRONMENT=prod -e SERVER_HOST=0.0.0.0 -e SERVER_PORT=3000 \
  -e LOG_LEVEL=INFO -e REQUEST_TIMEOUT=20 -e MAX_RESULTS=1 -e METRICS_ENABLED=true \
  your-image-name
```

## API Contract

### OpenAPI Documentation

FastAPI automatically generates interactive API documentation with Pydantic models. Access it at:

- **Swagger UI**: `http://127.0.0.1:3000/docs` — interactive API explorer
- **ReDoc**: `http://127.0.0.1:3000/redoc` — alternative documentation view

The `/docs` endpoint allows you to test the `/mcp` endpoint directly in your browser.

### JSON-RPC Endpoint: `/mcp`

The service implements the JSON-RPC 2.0 specification. It accepts `POST` requests and returns JSON-RPC responses.

#### Request format

```json
{
  "jsonrpc": "2.0",
  "method": "search",
  "params": {"query": "machine learning"},
  "id": 1
}
```

**Fields:**

- `jsonrpc`: always `"2.0"`
- `method`: `"search"` (other methods return "Method not found" error)
- `params`: object with `query` field (search term)
- `id`: unique request identifier for correlation

#### Success response

```json
{
  "jsonrpc": "2.0",
  "result": "## DuckDuckGo\n...\n\n## Wikidata\n...\n\n## ArXiv\n...",
  "id": 1
}
```

**Fields:**

- `result`: markdown-formatted aggregated results from all sources

#### Error response

```json
{
  "jsonrpc": "2.0",
  "error": {
    "code": -32600,
    "message": "Invalid Request"
  },
  "id": null
}
```

**Common error codes:**

- `-32700`: Parse error (malformed JSON)
- `-32600`: Invalid Request (missing/invalid fields)
- `-32601`: Method not found (unsupported method)
- `-32603`: Internal error (server failure)

### Pydantic Models

The service uses Pydantic for request/response validation and schema documentation:

- `SearchParams`: query string parameter definition
- `JSONRPCRequest`: complete request envelope
- `JSONRPCResponse`: success response with markdown result
- `JSONRPCError`: error object structure
- `JSONRPCErrorResponse`: error response envelope

These models are automatically documented in the OpenAPI schema and visible in the Swagger UI.

## Troubleshooting

- `ModuleNotFoundError`: ensure `src/` is on `PYTHONPATH` or run via `python main.py`
- `ConnectionError`: verify outbound internet access to DuckDuckGo, Wikidata, and ArXiv
- `TimeoutError`: increase `REQUEST_TIMEOUT`

## Notes

- The repository uses `src/` package layout for clean import boundaries.
- The default JSON-RPC method is `search`; other methods return `Method not found`.
- Metrics are exported in plaintext for Prometheus scraping.