Skip to main content
Glama
README.md
# MCP Server - DevOps Automation Platform

A comprehensive Model Context Protocol (MCP) server for DevOps automation with real Docker, Kubernetes, and AWS integrations, API authentication, Prometheus/Grafana monitoring, and LLM-powered natural language automation.

## Features

- **Real DevOps Integrations**
  - Docker container and image management
  - Kubernetes cluster operations (deployments, pods, services)
  - AWS EC2, S3, ECS, Lambda, RDS management
- **Security**
  - API key authentication
  - JWT token support
  - Rate limiting
- **Monitoring**
  - Prometheus metrics
  - Grafana dashboards
  - Health checks
  - Alertmanager integration
- **LLM Integration**
  - Ollama (local, free) - default
  - OpenAI API support
  - Natural language task automation

## Quick Start

### Option 1: Local Development

```bash
# Install dependencies
pip install -r requirements.txt

# Run the server
make dev
# or: uvicorn src.main_v2:app --reload --host 0.0.0.0 --port 8000

# Access the API docs
open http://localhost:8000/docs
```

### Option 2: Docker Compose (with Monitoring)

```bash
# Copy environment template
cp .env.example .env
# Edit .env with your credentials

# Start all services
make up
# or: docker-compose up -d

# Access services:
# - MCP Server: http://localhost:8000
# - Prometheus: http://localhost:9090
# - Grafana: http://localhost:3000 (admin/admin)
# - Alertmanager: http://localhost:9093
```

## Connect the DevOps Mastery Playbook MCP

This repository can expose the separate, read-only MCP server from a sibling `DevOps-Mastery-Playbook` checkout in VS Code. It is intentionally not merged into this server's infrastructure-operation tool adapter: the playbook MCP only searches and reads catalogued learning material and cannot run Docker, Kubernetes, cloud, shell, or Git operations.

Expected sibling layout:

```text
/home/th/
├── mcp-server/
└── DevOps-Mastery-Playbook/
```

Prepare the playbook MCP's isolated environment by following `tools/devops_mcp/README.md` in that checkout. Then open this repository in VS Code and enable the `devops-playbook` server from `.vscode/mcp.json`. The launcher defaults to `../DevOps-Mastery-Playbook`; for a different location, change `DEVOPS_PLAYBOOK_ROOT` in the MCP configuration. It checks for the catalog, server module, and executable playbook venv before starting, then uses stdio directly.

This configuration is local editor integration only. The existing HTTP DevOps API and its operational tools remain unchanged. Review their permissions separately; the read-only playbook server does not reduce the authority of this project's existing tools.

## API Endpoints

### Public Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/` | GET | Server info |
| `/health` | GET | Health check |
| `/metrics` | GET | Prometheus metrics |
| `/docs` | GET | API documentation |

### Authentication

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/auth/api-key` | POST | Generate API key |
| `/auth/token` | POST | Generate JWT token |

### Docker Operations

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/docker/containers` | GET | List containers |
| `/docker/containers/{action}` | POST | Start/stop/restart container |
| `/docker/containers/{id}/logs` | GET | Get container logs |
| `/docker/containers/{id}/stats` | GET | Get container stats |
| `/docker/images` | GET | List images |
| `/docker/images/pull` | POST | Pull an image |
| `/docker/system/info` | GET | Docker system info |
| `/docker/system/prune` | POST | Cleanup unused resources |

### Kubernetes Operations

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/k8s/cluster/info` | GET | Cluster info |
| `/k8s/nodes` | GET | List nodes |
| `/k8s/namespaces` | GET | List namespaces |
| `/k8s/pods` | GET | List pods |
| `/k8s/pods/{pod}/logs` | GET | Get pod logs |
| `/k8s/deployments` | GET | List deployments |
| `/k8s/deployments/scale` | POST | Scale deployment |
| `/k8s/deployments/restart` | POST | Restart deployment |
| `/k8s/deployments/rollback` | POST | Rollback deployment |
| `/k8s/deployments/image` | POST | Update image |
| `/k8s/services` | GET | List services |

### AWS Operations

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/aws/ec2/instances` | GET | List EC2 instances |
| `/aws/ec2/{action}` | POST | Start/stop/reboot EC2 |
| `/aws/s3/buckets` | GET | List S3 buckets |
| `/aws/s3/buckets/{bucket}/objects` | GET | List objects |
| `/aws/ecs/clusters` | GET | List ECS clusters |
| `/aws/lambda/functions` | GET | List Lambda functions |
| `/aws/lambda/invoke` | POST | Invoke Lambda |
| `/aws/rds/instances` | GET | List RDS instances |

### Task Automation

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/tasks/run` | POST | Run automation task (orchestrated/queue-friendly path, including `tailor_resume`) |
| `/resume/tailor` | POST | Tailor a base resume to a specific mission/job opening (direct synchronous path) |
| `/resume/tailor/upload` | POST | Tailor from uploaded PDF/CSV/TXT/MD and export as JSON/PDF/CSV/Markdown/DOCX |
| `/resume/exports/cleanup` | POST | Delete generated resume export file from temporary storage |
| `/deploy` | POST | Deploy service |
| `/cicd/trigger` | POST | Trigger CI/CD pipeline |
| `/logs` | GET | Get logs |
| `/alerts` | POST | Send alert |
| `/alerts/history` | GET | Alert history |

### Resume Tailoring Example

Use this endpoint to adapt a base resume to a specific mission or job opening.

#### Request Body

```json
{
  "base_resume": "# John Doe\n## Experience\n- Built CI/CD pipelines with GitHub Actions and Jenkins\n- Managed Kubernetes workloads on AWS EKS\n- Improved API latency by 35% in Python services",
  "job_description": "We need a DevOps engineer with Kubernetes, AWS, CI/CD, Python, observability, and production reliability experience.",
  "target_role": "Senior DevOps Engineer"
}
```

#### cURL (ready to run)

```bash
curl -X POST "http://localhost:8000/resume/tailor" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key" \
  -d '{
    "base_resume": "# John Doe\n## Experience\n- Built CI/CD pipelines with GitHub Actions and Jenkins\n- Managed Kubernetes workloads on AWS EKS\n- Improved API latency by 35% in Python services",
    "job_description": "We need a DevOps engineer with Kubernetes, AWS, CI/CD, Python, observability, and production reliability experience.",
    "target_role": "Senior DevOps Engineer"
  }'
```

#### cURL via Task Runner (ready to run)

```bash
curl -X POST "http://localhost:8000/tasks/run" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key" \
  -d '{
    "task_name": "tailor_resume",
    "parameters": {
      "base_resume": "# John Doe\n## Experience\n- Built CI/CD pipelines with GitHub Actions and Jenkins\n- Managed Kubernetes workloads on AWS EKS\n- Improved API latency by 35% in Python services",
      "job_description": "We need a DevOps engineer with Kubernetes, AWS, CI/CD, Python, observability, and production reliability experience.",
      "target_role": "Senior DevOps Engineer"
    }
  }'
```

Task runner with file inputs (PDF/CSV/TXT/MD paths on server):

```bash
curl -X POST "http://localhost:8000/tasks/run" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key" \
  -d '{
    "task_name": "tailor_resume",
    "parameters": {
      "base_resume_path": "/data/incoming/my-resume.pdf",
      "job_description_path": "/data/incoming/job-offer.csv",
      "target_role": "Senior Platform Engineer",
      "output_format": "docx",
      "docx_style": "resumeio_inspired",
      "use_v6_background": true
    }
  }'
```

When `output_format` is `pdf`, `csv`, `markdown`, or `docx`, `/tasks/run` returns an `output_file` path in the task result.
Supported `docx_style` values: `minimal_executive`, `modern_creative`, `online_clean`, `visualcv_inspired`, `resumeio_inspired`, `resumebuilder_inspired`.
Optional `use_v6_background` toggle (default `false`) applies the V6 soft-blue background and left accent band to any DOCX style.

#### Example Response

```json
{
  "success": true,
  "tailored_resume_markdown": "## Tailored Summary\nEngineer aligned to Senior DevOps Engineer, with experience evidenced in kubernetes, aws, ci, cd, python.\n\n## Highlighted Experience\n- Managed Kubernetes workloads on AWS EKS\n- Built CI/CD pipelines with GitHub Actions and Jenkins\n- Improved API latency by 35% in Python services",
  "keyword_alignment": {
    "matched": ["kubernetes", "aws", "python", "ci", "cd"],
    "partial": ["observability"],
    "missing": ["reliability"]
  },
  "gap_report": [
    "Missing or weak evidence for: reliability"
  ]
}
```

#### Which path should I use?

- Use `/resume/tailor` for a direct, synchronous API call when you only need resume tailoring.
- Use `/tasks/run` with `task_name: tailor_resume` when you want to route through the task orchestration flow (queueing, unified task handling, and automation pipelines).

#### File Upload + File Export

This endpoint accepts `PDF`, `CSV`, `TXT`, or `MD` files and can return output as `json`, `pdf`, `csv`, `markdown`, or `docx`.

```bash
curl -X POST "http://localhost:8000/resume/tailor/upload" \
  -H "X-API-Key: your-api-key" \
  -F "base_resume_file=@./my-resume.pdf" \
  -F "job_description_file=@./job-offer.csv" \
  -F "target_role=Senior Platform Engineer" \
  -F "output_format=docx" \
  -F "docx_style=resumebuilder_inspired" \
  -F "use_v6_background=true" \
  --output tailored_resume.docx
```

CSV export example:

```bash
curl -X POST "http://localhost:8000/resume/tailor/upload" \
  -H "X-API-Key: your-api-key" \
  -F "base_resume_file=@./my-resume.pdf" \
  -F "job_description_file=@./job-offer.txt" \
  -F "output_format=csv" \
  --output tailored_resume.csv
```

Cleanup generated export file (using `output_file` returned by task/upload flows):

```bash
curl -X POST "http://localhost:8000/resume/exports/cleanup" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key" \
  -d '{
    "output_file": "/tmp/tmpabcd1234.pdf"
  }'
```

Automatic TTL cleanup is enabled for generated exports:

- Job name: `resume_export_ttl_cleanup`
- Default schedule: every hour (`0 * * * *`)
- Default TTL: 24 hours
- Only generated files with prefix `resume_tailor_` and extensions `.pdf`, `.csv`, `.md` are deleted

Configuration via environment variables:

```bash
MCP_RESUME_EXPORT_TTL_HOURS=24
MCP_RESUME_EXPORT_CLEANUP_CRON="0 * * * *"
```

## Authentication

### Using API Key

```bash
# Generate an API key
curl -X POST "http://localhost:8000/auth/api-key?name=mykey"

# Use the key in requests
curl -H "X-API-Key: your-api-key" http://localhost:8000/docker/containers
```

### Using JWT Token

```bash
# Get a token
curl -X POST "http://localhost:8000/auth/token?user_id=admin"

# Use the token
curl -H "Authorization: Bearer your-token" http://localhost:8000/docker/containers
```

## LLM Integration (Natural Language)

### Ollama (Default - Local & Free)

```bash
# Start Ollama
ollama serve
ollama pull mistral

# Run LLM integration
python src/llm_integration.py
```

### OpenAI

```bash
export OPENAI_API_KEY="sk-..."
python src/llm_integration.py --openai
```

### Example Commands

```
👤 You: Check the health of our infrastructure
🤖 Assistant: The infrastructure is healthy. All systems operational.

👤 You: List all running Docker containers
🤖 Assistant: Found 3 running containers: nginx, redis, postgres

👤 You: Scale the web deployment to 5 replicas
🤖 Assistant: Scaled deployment "web" to 5 replicas successfully.

👤 You: Show me the last 50 lines of logs from the api pod
🤖 Assistant: Here are the recent logs...
```

## Environment Variables

```bash
# Authentication
MCP_API_KEY=your-api-key
MCP_JWT_SECRET=your-jwt-secret

# AWS
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret

# OpenAI (optional)
OPENAI_API_KEY=sk-...

# Grafana
GRAFANA_USER=admin
GRAFANA_PASSWORD=admin
```

## Project Structure

```
mcp/
├── src/
│   ├── main_v2.py           # Main server with all features
│   ├── main.py              # Basic server (legacy)
│   ├── auth.py              # Authentication module
│   ├── monitoring.py        # Prometheus metrics & health checks
│   ├── llm_integration.py   # Ollama/OpenAI integration
│   └── devops/
│       ├── docker_ops.py    # Docker operations
│       ├── kubernetes_ops.py # Kubernetes operations
│       └── aws_ops.py       # AWS operations
├── monitoring/
│   ├── prometheus.yml       # Prometheus config
│   ├── alertmanager.yml     # Alertmanager config
│   └── grafana/
│       ├── provisioning/    # Grafana provisioning
│       └── dashboards/      # Grafana dashboards
├── docs/
│   ├── llm_integration.md   # LLM guide
│   └── devops_endpoints.md  # Endpoint reference
├── .github/workflows/
│   └── ci.yml               # CI/CD pipeline
├── docker-compose.yml       # Full stack deployment
├── Dockerfile               # Container build
├── Makefile                 # Convenience commands
├── requirements.txt         # Python dependencies
└── .env.example             # Environment template
```

## Make Commands

```bash
make help         # Show all commands
make install      # Install dependencies
make dev          # Run development server
make run          # Run production server
make up           # Start all services (docker-compose)
make down         # Stop all services
make logs         # View logs
make test         # Run health check
make api-key      # Generate API key
make llm          # Run LLM integration
```


API Docs: http://localhost:8000/docs
Health: http://localhost:8000/health
Metrics: http://localhost:8000/metrics
Prometheus: http://localhost:9090 (when using docker-compose)
Grafana: http://localhost:3000 (admin/admin)

The server is running at http://localhost:8000 with all features active!

export MCP_API_KEY=$(curl -s -X POST "http://localhost:8000/auth/api-key?name=llm" | python -c "import sys,json; print(json.load(sys.stdin)['api_key'])") && python src/llm_integration.py

## Stopping the Server

```bash
# If running in foreground (started without &)
# Press Ctrl+C

# If running in background (started with &)
pkill -f "uvicorn.*8000"

# Or find the PID and kill it
lsof -i :8000
kill <PID>

# If using Docker Compose
make down
# or: docker-compose down
```

## Troubleshooting

### Common Issues and Solutions

#### 1. Server Not Responding / Hanging

**Symptoms:**
- `curl http://localhost:8000/health` hangs indefinitely
- API requests don't return
- Server appears to be running but unresponsive

**Solution:**
```bash
# Kill any stuck uvicorn processes
pkill -f "uvicorn.*8000"

# Restart the server
source /path/to/venv/bin/activate
uvicorn src.main_v2:app --host 0.0.0.0 --port 8000
```

#### 2. AttributeError: 'ScheduledJobRepository' object has no attribute 'list_all'

**Symptoms:**
- Server fails to start
- Error in startup event when loading scheduled jobs

**Solution:**
Ensure `src/database.py` has the `list_all()` method in `ScheduledJobRepository`:

```python
class ScheduledJobRepository:
    # ... other methods ...
    
    def list_all(self) -> List[Dict]:
        """List all scheduled jobs (alias for list method)."""
        return self.list()
```

#### 3. RuntimeWarning: coroutine 'Scheduler.start' was never awaited

**Symptoms:**
- Warning on server startup
- Scheduler not running properly
- Scheduled jobs not executing

**Solution:**
In `src/main_v2.py`, ensure async functions are awaited in startup/shutdown:

```python
@app.on_event("startup")
async def startup_event():
    await global_scheduler.start()  # Must use await
    # ... rest of startup

@app.on_event("shutdown")
async def shutdown_event():
    await global_scheduler.stop()   # Must use await
```

#### 4. LLM Integration Timeout / Long Response Times

**Symptoms:**
- LLM chat hangs or times out
- Very slow responses from Ollama
- `httpx.ReadTimeout` errors

**Solution:**
1. **Ollama Performance:** Ollama on CPU can be slow. Consider:
   ```bash
   # Use a smaller model
   ollama pull phi
   # Or use GPU if available
   ```

2. **Timeout Configuration:** The timeout is set to 300s in `llm_integration.py`. Adjust if needed:
   ```python
   timeout=httpx.Timeout(300.0)  # Increase for slower systems
   ```

3. **Log Truncation:** Large log outputs are automatically truncated to prevent token overflow.

#### 5. Port Already in Use

**Symptoms:**
- `Address already in use` error on startup
- Can't bind to port 8000

**Solution:**
```bash
# Find process using port 8000
lsof -i :8000

# Kill it
kill -9 <PID>
# or
pkill -f "uvicorn.*8000"
```

#### 6. Database Errors

**Symptoms:**
- `sqlite3.OperationalError: database is locked`
- Data not persisting

**Solution:**
```bash
# Check if multiple processes are accessing the DB
lsof mcp_data.db

# If locked, stop all server instances first
pkill -f "uvicorn"

# Then restart
uvicorn src.main_v2:app --host 0.0.0.0 --port 8000
```

#### 7. Missing MCP_API_KEY Environment Variable

**Symptoms:**
- LLM integration can't connect to MCP server
- 401 Unauthorized errors

**Solution:**
```bash
# Generate and export API key in one command
export MCP_API_KEY=$(curl -s -X POST "http://localhost:8000/auth/api-key?name=llm" | python -c "import sys,json; print(json.load(sys.stdin)['api_key'])")

# Verify it's set
echo $MCP_API_KEY

# Then run LLM integration
python src/llm_integration.py
```

### Health Check Commands

```bash
# Check if server is running
curl http://localhost:8000/health

# Check server info
curl http://localhost:8000/

# Check metrics
curl http://localhost:8000/metrics

# Test authentication
API_KEY=$(curl -s -X POST "http://localhost:8000/auth/api-key?name=test" | python -c "import sys,json; print(json.load(sys.stdin)['api_key'])")
curl -H "X-API-Key: $API_KEY" http://localhost:8000/docker/containers
```

### Logs Location

- **Server logs:** Console output from uvicorn
- **Audit logs:** SQLite database (`mcp_data.db` → `audit_logs` table)
- **Docker logs:** `docker-compose logs` (when using compose)

## License

MIT License