# Prometheus MCP Server
An MCP (Model Context Protocol) server for querying AWS Managed Prometheus (AMP) with SigV4 authentication. This server enables AI assistants to execute PromQL queries and discover metrics in a secure, VPC-isolated environment.
## Features
- **SigV4 Authentication**: Automatically signs requests using AWS credentials (supports EKS Pod Identity/IRSA)
- **5 MCP Tools**:
- `query_instant` - Execute instant PromQL queries
- `query_range` - Execute range queries for time series data
- `list_labels` - Get all label names
- `get_label_values` - Get values for a specific label
- `list_metrics` - Get all metric names with optional metadata
- **VPC Isolation**: Designed to run inside a VPC with no public exposure
- **Production Ready**: Includes Terraform, Kubernetes manifests, and comprehensive testing
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Customer VPC │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ EKS Cluster │ │
│ │ ┌─────────────────┐ ┌─────────────────────────────┐ │ │
│ │ │ Prometheus MCP │◄───│ ClusterIP Service :8080 │ │ │
│ │ │ (Pod Identity) │ │ (Internal Only) │ │ │
│ │ └────────┬────────┘ └─────────────────────────────┘ │ │
│ │ │ │ │
│ └───────────┼────────────────────────────────────────────────┘ │
│ │ SigV4 Signed HTTP │
│ ▼ │
│ ┌───────────────────────┐ │
│ │ AWS Managed Prometheus│ │
│ └───────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
## Quick Start
### Prerequisites
- Python 3.11+
- AWS CLI configured with credentials
- Docker (for building container images)
- Terraform 1.5+ (for infrastructure deployment)
- kubectl (for Kubernetes deployment)
- An SSH key pair in AWS
### Local Development
```bash
# Clone and install
cd prometheus-mcp
pip install -e ".[dev]"
# Set environment variables
export PROMETHEUS_WORKSPACE_ID="ws-your-workspace-id"
export AWS_REGION="us-east-1"
# Run the server
python -m prometheus_mcp.server
```
### Run Tests
```bash
# Install dev dependencies
pip install -e ".[dev]"
# Run unit tests
pytest tests/ -v
# Run with coverage
pytest tests/ --cov=prometheus_mcp --cov-report=html
```
## Production Deployment
### Step 1: Deploy Infrastructure with Terraform
```bash
cd deploy/terraform
# Initialize Terraform
terraform init
# Review the plan
terraform plan -var="ssh_key_name=your-key-name"
# Apply (creates VPC, EKS, AMP, ECR, Bastion)
terraform apply -var="ssh_key_name=your-key-name"
```
**Resources Created:**
- VPC with public/private subnets
- EKS cluster with managed node group
- AWS Managed Prometheus workspace
- ECR repository
- Bastion host for SSH access
- IAM roles with Pod Identity
### Step 2: Build and Push Docker Image
```bash
# Get ECR login command from terraform output
$(terraform output -raw docker_login_command)
# Build the image
docker build -t prometheus-mcp .
# Tag and push
ECR_URL=$(terraform output -raw ecr_repository_url)
docker tag prometheus-mcp:latest $ECR_URL:latest
docker push $ECR_URL:latest
```
### Step 3: Deploy to Kubernetes
```bash
# Update kubeconfig
$(terraform output -raw kubeconfig_command)
# Get values for K8s manifests
export ECR_URL=$(terraform output -raw ecr_repository_url)
export WORKSPACE_ID=$(terraform output -raw amp_workspace_id)
export ROLE_ARN=$(terraform output -raw prometheus_mcp_role_arn)
# Update manifests with actual values
sed -i '' "s|\${ECR_URL}|$ECR_URL|g" deploy/k8s/deployment.yaml
sed -i '' "s|\${WORKSPACE_ID}|$WORKSPACE_ID|g" deploy/k8s/deployment.yaml
sed -i '' "s|\${PROMETHEUS_MCP_ROLE_ARN}|$ROLE_ARN|g" deploy/k8s/service-account.yaml
# Apply manifests
kubectl apply -f deploy/k8s/
```
### Step 4: Verify Deployment
```bash
# Check pods are running
kubectl get pods -n prometheus-mcp
# Check logs
kubectl logs -n prometheus-mcp -l app=prometheus-mcp
```
## Testing via SSH Tunnel
Since the MCP server is only accessible within the VPC, use SSH tunneling to test from your laptop.
### Method 1: Manual Setup (3 Terminals)
**Terminal 1 - SSH to Bastion and Port-Forward:**
```bash
# SSH to bastion
ssh -i ~/.ssh/your-key.pem ec2-user@<BASTION_IP>
# On the bastion, set up kubectl
~/setup-kubectl.sh
# Start port-forward
~/port-forward-mcp.sh
```
**Terminal 2 - SSH Tunnel:**
```bash
ssh -i ~/.ssh/your-key.pem -L 8080:localhost:8080 ec2-user@<BASTION_IP>
```
**Terminal 3 - MCP Inspector:**
```bash
npx @anthropic/mcp-inspector http://localhost:8080
```
### Method 2: Using the Test Script
```bash
cd deploy/scripts
./test-via-tunnel.sh
```
This script will display all the commands you need to run.
### Verify VPC Isolation
```bash
# This should FAIL (timeout) - proves VPC isolation
kubectl get pod -n prometheus-mcp -o jsonpath='{.items[0].status.podIP}'
curl http://<POD_IP>:8080/health --connect-timeout 5
# Expected: Connection timed out
# This should SUCCEED (via SSH tunnel)
curl http://localhost:8080/health
# Expected: {"status": "healthy"}
```
## MCP Tools Reference
### query_instant
Execute an instant PromQL query at a single point in time.
```json
{
"name": "query_instant",
"arguments": {
"query": "up",
"time": "2024-01-15T10:00:00Z" // optional
}
}
```
### query_range
Execute a range query to get time series data.
```json
{
"name": "query_range",
"arguments": {
"query": "rate(http_requests_total[5m])",
"start": "2024-01-15T00:00:00Z",
"end": "2024-01-15T12:00:00Z",
"step": "1m"
}
}
```
### list_labels
Get all label names.
```json
{
"name": "list_labels",
"arguments": {
"match": ["up", "http_requests_total"] // optional
}
}
```
### get_label_values
Get all values for a specific label.
```json
{
"name": "get_label_values",
"arguments": {
"label_name": "job",
"match": ["{namespace='production'}"] // optional
}
}
```
### list_metrics
Get all metric names.
```json
{
"name": "list_metrics",
"arguments": {
"with_metadata": true // optional, slower but includes type/help/unit
}
}
```
## Configuration
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `PROMETHEUS_WORKSPACE_ID` | AMP workspace ID (required) | - |
| `AWS_REGION` | AWS region | `us-east-1` |
### Terraform Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `aws_region` | AWS region | `us-east-1` |
| `ssh_key_name` | SSH key pair name (required) | - |
| `allowed_ssh_cidr` | CIDR for SSH access | `0.0.0.0/0` |
| `eks_cluster_version` | Kubernetes version | `1.29` |
| `eks_node_instance_type` | EKS node instance type | `t3.medium` |
| `bastion_instance_type` | Bastion instance type | `t3.micro` |
## Project Structure
```
prometheus-mcp/
├── prometheus_mcp/
│ ├── __init__.py
│ ├── server.py # MCP server entry point
│ ├── amp_client.py # SigV4-authenticated AMP client
│ └── promql/
│ ├── __init__.py
│ ├── models.py # Pydantic response models
│ └── tools.py # MCP tool implementations
├── deploy/
│ ├── terraform/ # Infrastructure as Code
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ ├── outputs.tf
│ │ ├── vpc.tf
│ │ ├── eks.tf
│ │ ├── amp.tf
│ │ ├── ecr.tf
│ │ ├── iam.tf
│ │ └── bastion.tf
│ ├── k8s/ # Kubernetes manifests
│ │ ├── namespace.yaml
│ │ ├── service-account.yaml
│ │ ├── deployment.yaml
│ │ └── service.yaml
│ └── scripts/
│ ├── push-sample-metrics.sh
│ ├── test-via-tunnel.sh
│ └── cleanup.sh
├── tests/
│ ├── conftest.py
│ ├── test_amp_client.py
│ └── test_promql_tools.py
├── Dockerfile
├── pyproject.toml
└── README.md
```
## Cleanup
To destroy all resources:
```bash
cd deploy/scripts
./cleanup.sh
```
Or manually:
```bash
# Delete K8s resources
kubectl delete -f deploy/k8s/
# Destroy Terraform infrastructure
cd deploy/terraform
terraform destroy
```
## Security Considerations
1. **VPC Isolation**: The MCP server is only accessible via ClusterIP service within the EKS cluster
2. **Pod Identity (IRSA)**: Uses AWS IAM roles for service accounts instead of static credentials
3. **Least Privilege**: IAM role only has permissions to query AMP, not write
4. **No Public Endpoints**: All access is via SSH tunnel through bastion
5. **Container Security**: Runs as non-root user with read-only filesystem
## Troubleshooting
### Pod not starting
```bash
kubectl describe pod -n prometheus-mcp -l app=prometheus-mcp
kubectl logs -n prometheus-mcp -l app=prometheus-mcp
```
### IAM permissions issues
```bash
# Verify the service account has the correct annotation
kubectl get sa -n prometheus-mcp prometheus-mcp -o yaml
# Check if Pod Identity is working
kubectl exec -n prometheus-mcp -it <pod-name> -- aws sts get-caller-identity
```
### Cannot connect via SSH tunnel
```bash
# Verify bastion is running
aws ec2 describe-instances --filters "Name=tag:Name,Values=prometheus-mcp-bastion" --query "Reservations[].Instances[].State.Name"
# Check security group allows SSH
aws ec2 describe-security-groups --group-ids <bastion-sg-id>
```
## License
MIT
