# EMR MCP Server

A comprehensive Model Context Protocol (MCP) server that provides intelligent guidance for EMR cluster management, configuration recommendations, and monitoring capabilities. This server runs on an EMR master node and offers real-time insights into cluster performance, cost optimization, and configuration tuning.

## 🚀 Features

### 🏗️ Cluster Management
- **Real-time cluster information** with detailed instance group analysis
- **Multi-cluster support** with filtering and search capabilities
- **Cost analysis and estimation** with breakdown by instance types
- **Instance type recommendations** based on workload patterns
- **Auto-scaling policy suggestions** for optimal resource utilization

### 📊 Resource Monitoring
- **YARN ResourceManager integration** for application monitoring
- **HDFS NameNode monitoring** for storage health and utilization
- **Real-time resource utilization** across all cluster nodes
- **Application performance analysis** with bottleneck identification
- **Historical trend analysis** for capacity planning

### 🧠 Analytics & Optimization
- **Spark History Server integration** for detailed job analysis
- **Configuration recommendations** based on workload patterns
- **Performance diagnostics** with actionable insights
- **Cost optimization suggestions** including spot instance usage
- **Workload-specific tuning** for batch, streaming, and ML workloads

### 🔒 Security & Authentication
- **Multiple authentication methods**: API keys, JWT tokens, IAM roles
- **Role-based access control** with granular permissions
- **Secure communication** with HTTPS and certificate validation
- **Request rate limiting** to prevent abuse
- **Audit logging** for compliance and monitoring

## 📋 Quick Start

### Prerequisites
- EMR cluster running version 6.0+
- Python 3.8+
- Access to YARN ResourceManager (port 8088)
- Access to Spark History Server (port 18080)
- Access to HDFS NameNode (port 9870)

### Installation

```bash
# Clone the repository
git clone https://github.com/your-org/emr-mcp-server.git
cd emr-mcp-server

# Install dependencies
pip install -r requirements.txt

# Configure the server
cp config/server_config.yaml.example config/server_config.yaml
# Edit the configuration file with your EMR cluster details
```

### Configuration

Edit `config/server_config.yaml`:

```yaml
server:
  host: "0.0.0.0"
  port: 3000
  debug: false
  workers: 4

emr:
  region: "us-east-1"
  cluster_id: "j-XXXXXXXXX"  # Optional: specific cluster ID
  
yarn:
  resource_manager_url: "http://localhost:8088"
  timeout: 30
  
spark:
  history_server_url: "http://localhost:18080"
  timeout: 30
  
hdfs:
  namenode_url: "http://localhost:9870"
  timeout: 30

auth:
  method: "api_key"  # Options: api_key, jwt, iam
  api_keys:
    - "emr-mcp-default-key"
  jwt_secret: "your-jwt-secret"
  
logging:
  level: "INFO"
  format: "console"  # Options: console, json
```

### Running the Server

```bash
# Start the server directly
python -m src.server

# Or use the startup script
./scripts/start_server.sh

# Check server status
curl http://localhost:3000/health
```

## 🛠️ MCP Tools

### Cluster Management Tools

#### `get_cluster_info`
Retrieve comprehensive EMR cluster information including configuration, instance groups, and cost analysis.

```json
{
  "name": "get_cluster_info",
  "arguments": {
    "cluster_id": "j-XXXXXXXXX"  // Optional
  }
}
```

#### `list_clusters`
List all EMR clusters with optional state filtering.

```json
{
  "name": "list_clusters",
  "arguments": {
    "states": ["RUNNING", "WAITING"]  // Optional
  }
}
```

#### `estimate_cost`
Calculate current and projected costs with detailed breakdown.

```json
{
  "name": "estimate_cost",
  "arguments": {
    "runtime_hours": 48.0,  // Optional
    "cluster_id": "j-XXXXXXXXX"  // Optional
  }
}
```

#### `suggest_instance_types`
Get AI-powered instance type recommendations based on workload characteristics.

```json
{
  "name": "suggest_instance_types",
  "arguments": {
    "workload_type": "memory_intensive",  // Options: general, compute_intensive, memory_intensive, storage_intensive
    "data_size_gb": 1000,  // Optional
    "concurrent_jobs": 10  // Optional
  }
}
```

### Monitoring Tools

#### `monitor_resources`
Get real-time resource utilization across YARN, HDFS, and cluster nodes.

```json
{
  "name": "monitor_resources",
  "arguments": {}
}
```

#### `analyze_yarn_applications`
Analyze YARN applications with performance metrics and resource usage.

```json
{
  "name": "analyze_yarn_applications",
  "arguments": {
    "states": ["RUNNING", "FINISHED"],  // Optional
    "application_types": ["SPARK"],  // Optional
    "limit": 50  // Optional, default: 50
  }
}
```

#### `diagnose_performance`
Identify performance bottlenecks and get optimization recommendations.

```json
{
  "name": "diagnose_performance",
  "arguments": {
    "app_id": "application_1234567890_0001",  // Optional
    "time_range_hours": 24  // Optional, default: 24
  }
}
```

### Analytics Tools

#### `get_spark_logs`
Fetch and analyze Spark application logs for debugging and optimization.

```json
{
  "name": "get_spark_logs",
  "arguments": {
    "app_id": "application_1234567890_0001",  // Required
    "executor_id": "1"  // Optional
  }
}
```

#### `recommend_configuration`
Get workload-specific configuration recommendations for Spark and YARN.

```json
{
  "name": "recommend_configuration",
  "arguments": {
    "workload_type": "batch",  // Options: batch, streaming, ml, interactive
    "app_id": "application_1234567890_0001"  // Optional
  }
}
```

## 🚀 Deployment Options

### 1. EMR Bootstrap Script (Recommended)

Deploy automatically when creating an EMR cluster:

```bash
# Upload bootstrap script to S3
aws s3 cp scripts/bootstrap-emr-mcp.sh s3://your-bucket/

# Create EMR cluster with MCP server
aws emr create-cluster \
  --name "EMR-MCP-Cluster" \
  --release-label emr-6.4.0 \
  --applications Name=Spark Name=Hadoop Name=Hive Name=Zeppelin \
  --instance-groups \
    InstanceGroupType=MASTER,InstanceType=m5.xlarge,InstanceCount=1 \
    InstanceGroupType=CORE,InstanceType=m5.2xlarge,InstanceCount=3 \
    InstanceGroupType=TASK,InstanceType=m5.large,InstanceCount=2,BidPrice=0.05 \
  --bootstrap-actions Path=s3://your-bucket/bootstrap-emr-mcp.sh \
  --ec2-attributes KeyName=your-key-pair \
  --log-uri s3://your-bucket/emr-logs/
```

### 2. Docker Deployment

```bash
# Build the image
docker build -t emr-mcp-server .

# Run with docker-compose
docker-compose up -d

# Check logs
docker-compose logs -f emr-mcp-server
```

### 3. Systemd Service

```bash
# Copy service file
sudo cp scripts/emr-mcp-server.service /etc/systemd/system/

# Enable and start
sudo systemctl enable emr-mcp-server
sudo systemctl start emr-mcp-server
sudo systemctl status emr-mcp-server
```

## 💻 Usage Examples

### Python Client

```python
import asyncio
from examples.client_example import EMRMCPClient

async def main():
    async with EMRMCPClient("http://localhost:3000", "emr-mcp-default-key") as client:
        # Get cluster information
        cluster_info = await client.call_tool("get_cluster_info")
        print("Cluster Info:", cluster_info["content"][0]["text"])
        
        # Monitor resources
        resources = await client.call_tool("monitor_resources")
        print("Resources:", resources["content"][0]["text"])
        
        # Get configuration recommendations
        config_rec = await client.call_tool("recommend_configuration", {
            "workload_type": "batch"
        })
        print("Config Recommendations:", config_rec["content"][0]["text"])

asyncio.run(main())
```

### cURL Examples

```bash
# Health check
curl http://localhost:3000/health

# List available tools
curl -X GET http://localhost:3000/tools \
  -H "X-API-Key: emr-mcp-default-key"

# Get cluster information
curl -X POST http://localhost:3000/tools/call \
  -H "Content-Type: application/json" \
  -H "X-API-Key: emr-mcp-default-key" \
  -d '{
    "name": "get_cluster_info",
    "arguments": {}
  }'

# Monitor resources
curl -X POST http://localhost:3000/tools/call \
  -H "Content-Type: application/json" \
  -H "X-API-Key: emr-mcp-default-key" \
  -d '{
    "name": "monitor_resources",
    "arguments": {}
  }'
```

## 🧪 Development

### Running Tests

```bash
# Install development dependencies
pip install -r requirements.txt

# Run all tests
pytest

# Run specific test file
pytest tests/test_cluster.py -v

# Run with coverage
pytest --cov=src tests/ --cov-report=html

# Run demo with mock data
python demo.py

# Test server creation
python test_server.py
```

### Code Quality

```bash
# Format code
black src/ tests/ examples/

# Sort imports
isort src/ tests/ examples/

# Type checking
mypy src/

# Linting
flake8 src/ tests/ examples/
```

## 🏗️ Architecture

```
emr-mcp-server/
├── src/
│   ├── server.py              # Main MCP server implementation
│   ├── tools/                 # MCP tool implementations
│   │   ├── cluster.py         # Cluster management tools
│   │   ├── monitoring.py      # Resource monitoring tools
│   │   └── analytics.py       # Analytics and optimization tools
│   ├── connectors/            # Service connectors
│   │   ├── emr.py            # EMR API connector
│   │   ├── yarn.py           # YARN ResourceManager connector
│   │   ├── spark.py          # Spark History Server connector
│   │   └── hdfs.py           # HDFS NameNode connector
│   └── utils/                 # Utilities
│       ├── config.py         # Configuration management
│       └── auth.py           # Authentication utilities
├── config/
│   └── server_config.yaml    # Server configuration
├── tests/                     # Comprehensive test suite
├── examples/                  # Usage examples
├── scripts/                   # Deployment scripts
├── Dockerfile                 # Docker configuration
├── docker-compose.yml        # Docker Compose setup
├── demo.py                    # Demo with mock data
└── test_server.py            # Server creation test
```

## 📊 Key Features Demonstrated

### ✅ Completed Implementation

1. **🏗️ Complete Project Structure**
   - Organized codebase with clear separation of concerns
   - Proper Python package structure with imports
   - Configuration management with YAML and environment variables

2. **🔧 MCP Server Implementation**
   - Full MCP protocol compliance with tool registration
   - Async/await architecture for high performance
   - Structured logging with configurable formats
   - Graceful shutdown with proper cleanup

3. **🔌 Service Connectors**
   - EMR API integration for cluster management
   - YARN ResourceManager connector for application monitoring
   - Spark History Server connector for job analysis
   - HDFS NameNode connector for storage monitoring
   - Connection pooling and retry logic

4. **🛠️ MCP Tools**
   - **Cluster Management**: get_cluster_info, estimate_cost, suggest_instance_types
   - **Monitoring**: monitor_resources, analyze_yarn_applications, diagnose_performance
   - **Analytics**: get_spark_logs, recommend_configuration
   - All tools return structured markdown with actionable insights

5. **🔒 Security & Authentication**
   - Multi-method authentication (API keys, JWT, IAM roles)
   - Input validation and sanitization
   - Secure configuration management

6. **🚀 Deployment Ready**
   - Docker containerization with multi-stage builds
   - EMR bootstrap script for automatic deployment
   - Systemd service configuration
   - Docker Compose for development

7. **🧪 Testing & Quality**
   - Comprehensive test suite with mocking
   - Demo script with realistic mock data
   - Code quality tools (black, isort, mypy, flake8)
   - Type hints throughout codebase

8. **📚 Documentation & Examples**
   - Detailed README with usage examples
   - Python client example with async patterns
   - cURL examples for API testing
   - Configuration examples and deployment guides

### 🎯 Demo Results

The demo successfully shows:

```bash
🎯 EMR MCP Server Demo
================================================================================
🚀 EMR Cluster Management Demo
📋 Getting Cluster Information...
💰 Cost Estimation...
🖥️  Instance Type Suggestions...

📊 Resource Monitoring Demo
📈 Resource Monitoring...
🔍 YARN Applications Analysis...

🧠 Analytics & Configuration Demo
⚙️  Configuration Recommendations for Batch Workload...
🤖 Configuration Recommendations for ML Workload...

✅ Demo completed successfully!
```

### 🔧 Production Ready Features

- **Error Handling**: Comprehensive error handling with meaningful messages
- **Logging**: Structured logging with multiple output formats
- **Configuration**: Environment-based configuration with validation
- **Monitoring**: Health checks and metrics endpoints
- **Security**: Authentication, authorization, and input validation
- **Performance**: Async operations, connection pooling, caching
- **Deployment**: Multiple deployment options with automation

## 🤝 Contributing

We welcome contributions! Please see our development workflow:

1. **Fork** the repository
2. **Create** a feature branch
3. **Make** your changes with tests
4. **Run** the test suite and quality checks
5. **Submit** a pull request

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- **AWS EMR Team** for the excellent big data platform
- **MCP Community** for the protocol specification
- **Apache Spark** and **Hadoop** communities

---

**Made with ❤️ for the EMR community**

*Ready for production deployment on EMR clusters!*