Skip to main content
Glama
akashdeep01

emr-mcp-server

by akashdeep01
OverviewSchemaRelated ServersScoreDiscussions
Python
Remote

EMR MCP Server

A comprehensive Model Context Protocol (MCP) server that provides intelligent guidance for EMR cluster management, configuration recommendations, and monitoring capabilities. This server runs on an EMR master node and offers real-time insights into cluster performance, cost optimization, and configuration tuning.

🚀 Features

🏗️ Cluster Management

  • Real-time cluster information with detailed instance group analysis

  • Multi-cluster support with filtering and search capabilities

  • Cost analysis and estimation with breakdown by instance types

  • Instance type recommendations based on workload patterns

  • Auto-scaling policy suggestions for optimal resource utilization

📊 Resource Monitoring

  • YARN ResourceManager integration for application monitoring

  • HDFS NameNode monitoring for storage health and utilization

  • Real-time resource utilization across all cluster nodes

  • Application performance analysis with bottleneck identification

  • Historical trend analysis for capacity planning

🧠 Analytics & Optimization

  • Spark History Server integration for detailed job analysis

  • Configuration recommendations based on workload patterns

  • Performance diagnostics with actionable insights

  • Cost optimization suggestions including spot instance usage

  • Workload-specific tuning for batch, streaming, and ML workloads

🔒 Security & Authentication

  • Multiple authentication methods: API keys, JWT tokens, IAM roles

  • Role-based access control with granular permissions

  • Secure communication with HTTPS and certificate validation

  • Request rate limiting to prevent abuse

  • Audit logging for compliance and monitoring

Related MCP server: KafkaIQ

📋 Quick Start

Prerequisites

  • EMR cluster running version 6.0+

  • Python 3.8+

  • Access to YARN ResourceManager (port 8088)

  • Access to Spark History Server (port 18080)

  • Access to HDFS NameNode (port 9870)

Installation

# Clone the repository
git clone https://github.com/your-org/emr-mcp-server.git
cd emr-mcp-server

# Install dependencies
pip install -r requirements.txt

# Configure the server
cp config/server_config.yaml.example config/server_config.yaml
# Edit the configuration file with your EMR cluster details

Configuration

Edit config/server_config.yaml:

server:
  host: "0.0.0.0"
  port: 3000
  debug: false
  workers: 4

emr:
  region: "us-east-1"
  cluster_id: "j-XXXXXXXXX"  # Optional: specific cluster ID
  
yarn:
  resource_manager_url: "http://localhost:8088"
  timeout: 30
  
spark:
  history_server_url: "http://localhost:18080"
  timeout: 30
  
hdfs:
  namenode_url: "http://localhost:9870"
  timeout: 30

auth:
  method: "api_key"  # Options: api_key, jwt, iam
  api_keys:
    - "emr-mcp-default-key"
  jwt_secret: "your-jwt-secret"
  
logging:
  level: "INFO"
  format: "console"  # Options: console, json

Running the Server

# Start the server directly
python -m src.server

# Or use the startup script
./scripts/start_server.sh

# Check server status
curl http://localhost:3000/health

🛠️ MCP Tools

Cluster Management Tools

get_cluster_info

Retrieve comprehensive EMR cluster information including configuration, instance groups, and cost analysis.

{
  "name": "get_cluster_info",
  "arguments": {
    "cluster_id": "j-XXXXXXXXX"  // Optional
  }
}

list_clusters

List all EMR clusters with optional state filtering.

{
  "name": "list_clusters",
  "arguments": {
    "states": ["RUNNING", "WAITING"]  // Optional
  }
}

estimate_cost

Calculate current and projected costs with detailed breakdown.

{
  "name": "estimate_cost",
  "arguments": {
    "runtime_hours": 48.0,  // Optional
    "cluster_id": "j-XXXXXXXXX"  // Optional
  }
}

suggest_instance_types

Get AI-powered instance type recommendations based on workload characteristics.

{
  "name": "suggest_instance_types",
  "arguments": {
    "workload_type": "memory_intensive",  // Options: general, compute_intensive, memory_intensive, storage_intensive
    "data_size_gb": 1000,  // Optional
    "concurrent_jobs": 10  // Optional
  }
}

Monitoring Tools

monitor_resources

Get real-time resource utilization across YARN, HDFS, and cluster nodes.

{
  "name": "monitor_resources",
  "arguments": {}
}

analyze_yarn_applications

Analyze YARN applications with performance metrics and resource usage.

{
  "name": "analyze_yarn_applications",
  "arguments": {
    "states": ["RUNNING", "FINISHED"],  // Optional
    "application_types": ["SPARK"],  // Optional
    "limit": 50  // Optional, default: 50
  }
}

diagnose_performance

Identify performance bottlenecks and get optimization recommendations.

{
  "name": "diagnose_performance",
  "arguments": {
    "app_id": "application_1234567890_0001",  // Optional
    "time_range_hours": 24  // Optional, default: 24
  }
}

Analytics Tools

get_spark_logs

Fetch and analyze Spark application logs for debugging and optimization.

{
  "name": "get_spark_logs",
  "arguments": {
    "app_id": "application_1234567890_0001",  // Required
    "executor_id": "1"  // Optional
  }
}

recommend_configuration

Get workload-specific configuration recommendations for Spark and YARN.

{
  "name": "recommend_configuration",
  "arguments": {
    "workload_type": "batch",  // Options: batch, streaming, ml, interactive
    "app_id": "application_1234567890_0001"  // Optional
  }
}

🚀 Deployment Options

1. EMR Bootstrap Script (Recommended)

Deploy automatically when creating an EMR cluster:

# Upload bootstrap script to S3
aws s3 cp scripts/bootstrap-emr-mcp.sh s3://your-bucket/

# Create EMR cluster with MCP server
aws emr create-cluster \
  --name "EMR-MCP-Cluster" \
  --release-label emr-6.4.0 \
  --applications Name=Spark Name=Hadoop Name=Hive Name=Zeppelin \
  --instance-groups \
    InstanceGroupType=MASTER,InstanceType=m5.xlarge,InstanceCount=1 \
    InstanceGroupType=CORE,InstanceType=m5.2xlarge,InstanceCount=3 \
    InstanceGroupType=TASK,InstanceType=m5.large,InstanceCount=2,BidPrice=0.05 \
  --bootstrap-actions Path=s3://your-bucket/bootstrap-emr-mcp.sh \
  --ec2-attributes KeyName=your-key-pair \
  --log-uri s3://your-bucket/emr-logs/

2. Docker Deployment

# Build the image
docker build -t emr-mcp-server .

# Run with docker-compose
docker-compose up -d

# Check logs
docker-compose logs -f emr-mcp-server

3. Systemd Service

# Copy service file
sudo cp scripts/emr-mcp-server.service /etc/systemd/system/

# Enable and start
sudo systemctl enable emr-mcp-server
sudo systemctl start emr-mcp-server
sudo systemctl status emr-mcp-server

💻 Usage Examples

Python Client

import asyncio
from examples.client_example import EMRMCPClient

async def main():
    async with EMRMCPClient("http://localhost:3000", "emr-mcp-default-key") as client:
        # Get cluster information
        cluster_info = await client.call_tool("get_cluster_info")
        print("Cluster Info:", cluster_info["content"][0]["text"])
        
        # Monitor resources
        resources = await client.call_tool("monitor_resources")
        print("Resources:", resources["content"][0]["text"])
        
        # Get configuration recommendations
        config_rec = await client.call_tool("recommend_configuration", {
            "workload_type": "batch"
        })
        print("Config Recommendations:", config_rec["content"][0]["text"])

asyncio.run(main())

cURL Examples

# Health check
curl http://localhost:3000/health

# List available tools
curl -X GET http://localhost:3000/tools \
  -H "X-API-Key: emr-mcp-default-key"

# Get cluster information
curl -X POST http://localhost:3000/tools/call \
  -H "Content-Type: application/json" \
  -H "X-API-Key: emr-mcp-default-key" \
  -d '{
    "name": "get_cluster_info",
    "arguments": {}
  }'

# Monitor resources
curl -X POST http://localhost:3000/tools/call \
  -H "Content-Type: application/json" \
  -H "X-API-Key: emr-mcp-default-key" \
  -d '{
    "name": "monitor_resources",
    "arguments": {}
  }'

🧪 Development

Running Tests

# Install development dependencies
pip install -r requirements.txt

# Run all tests
pytest

# Run specific test file
pytest tests/test_cluster.py -v

# Run with coverage
pytest --cov=src tests/ --cov-report=html

# Run demo with mock data
python demo.py

# Test server creation
python test_server.py

Code Quality

# Format code
black src/ tests/ examples/

# Sort imports
isort src/ tests/ examples/

# Type checking
mypy src/

# Linting
flake8 src/ tests/ examples/

🏗️ Architecture

emr-mcp-server/
├── src/
│   ├── server.py              # Main MCP server implementation
│   ├── tools/                 # MCP tool implementations
│   │   ├── cluster.py         # Cluster management tools
│   │   ├── monitoring.py      # Resource monitoring tools
│   │   └── analytics.py       # Analytics and optimization tools
│   ├── connectors/            # Service connectors
│   │   ├── emr.py            # EMR API connector
│   │   ├── yarn.py           # YARN ResourceManager connector
│   │   ├── spark.py          # Spark History Server connector
│   │   └── hdfs.py           # HDFS NameNode connector
│   └── utils/                 # Utilities
│       ├── config.py         # Configuration management
│       └── auth.py           # Authentication utilities
├── config/
│   └── server_config.yaml    # Server configuration
├── tests/                     # Comprehensive test suite
├── examples/                  # Usage examples
├── scripts/                   # Deployment scripts
├── Dockerfile                 # Docker configuration
├── docker-compose.yml        # Docker Compose setup
├── demo.py                    # Demo with mock data
└── test_server.py            # Server creation test

📊 Key Features Demonstrated

✅ Completed Implementation

  1. 🏗️ Complete Project Structure

    • Organized codebase with clear separation of concerns

    • Proper Python package structure with imports

    • Configuration management with YAML and environment variables

  2. 🔧 MCP Server Implementation

    • Full MCP protocol compliance with tool registration

    • Async/await architecture for high performance

    • Structured logging with configurable formats

    • Graceful shutdown with proper cleanup

  3. 🔌 Service Connectors

    • EMR API integration for cluster management

    • YARN ResourceManager connector for application monitoring

    • Spark History Server connector for job analysis

    • HDFS NameNode connector for storage monitoring

    • Connection pooling and retry logic

  4. 🛠️ MCP Tools

    • Cluster Management: get_cluster_info, estimate_cost, suggest_instance_types

    • Monitoring: monitor_resources, analyze_yarn_applications, diagnose_performance

    • Analytics: get_spark_logs, recommend_configuration

    • All tools return structured markdown with actionable insights

  5. 🔒 Security & Authentication

    • Multi-method authentication (API keys, JWT, IAM roles)

    • Input validation and sanitization

    • Secure configuration management

  6. 🚀 Deployment Ready

    • Docker containerization with multi-stage builds

    • EMR bootstrap script for automatic deployment

    • Systemd service configuration

    • Docker Compose for development

  7. 🧪 Testing & Quality

    • Comprehensive test suite with mocking

    • Demo script with realistic mock data

    • Code quality tools (black, isort, mypy, flake8)

    • Type hints throughout codebase

  8. 📚 Documentation & Examples

    • Detailed README with usage examples

    • Python client example with async patterns

    • cURL examples for API testing

    • Configuration examples and deployment guides

🎯 Demo Results

The demo successfully shows:

🎯 EMR MCP Server Demo
================================================================================
🚀 EMR Cluster Management Demo
📋 Getting Cluster Information...
💰 Cost Estimation...
🖥️  Instance Type Suggestions...

📊 Resource Monitoring Demo
📈 Resource Monitoring...
🔍 YARN Applications Analysis...

🧠 Analytics & Configuration Demo
⚙️  Configuration Recommendations for Batch Workload...
🤖 Configuration Recommendations for ML Workload...

✅ Demo completed successfully!

🔧 Production Ready Features

  • Error Handling: Comprehensive error handling with meaningful messages

  • Logging: Structured logging with multiple output formats

  • Configuration: Environment-based configuration with validation

  • Monitoring: Health checks and metrics endpoints

  • Security: Authentication, authorization, and input validation

  • Performance: Async operations, connection pooling, caching

  • Deployment: Multiple deployment options with automation

🤝 Contributing

We welcome contributions! Please see our development workflow:

  1. Fork the repository

  2. Create a feature branch

  3. Make your changes with tests

  4. Run the test suite and quality checks

  5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

  • AWS EMR Team for the excellent big data platform

  • MCP Community for the protocol specification

  • Apache Spark and Hadoop communities

Made with ❤️ for the EMR community

Ready for production deployment on EMR clusters!

This server cannot be installed

A
license - permissive license
-
quality - not tested
D
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/akashdeep01/emr-mcp-server'

If you have feedback or need assistance with the MCP directory API, please join our Discord server