MCP SSH Orchestrator

# Security Guide This document provides security guidance for deploying and operating mcp-ssh-orchestrator in production environments. ## Threat Model ### Assets - **SSH Private Keys**: Used for authentication to target hosts - **Credentials**: Passwords, passphrases, and other secrets - **Target Hosts**: SSH servers under management - **Command History**: Audit logs of executed commands - **Configuration**: Policy rules and host inventory ### Threats 1. **Unauthorized Command Execution**: Attacker gains ability to run arbitrary commands 2. **Credential Theft**: SSH keys or passwords are exposed 3. **Lateral Movement**: Attacker pivots from compromised host to others 4. **Data Exfiltration**: Sensitive output is leaked 5. **Denial of Service**: Resource exhaustion or service disruption 6. **Policy Bypass**: Circumventing allow/deny rules ### Mitigations All threats are addressed through defense-in-depth controls documented below. ## Authentication & Credentials ### Path Traversal Protection **Security Enhancement:** All credential and key path resolution includes path traversal protection to prevent unauthorized file access. #### Secret Path Protection Secret names and paths are validated to prevent directory traversal attacks: 1. **Secret Name Validation**: Only alphanumeric characters, dashes, and underscores are allowed ```yaml # ✅ Valid secret names password_secret: "prod_password" password_secret: "key-passphrase-2024" password_secret: "admin_password_1" # ❌ Invalid (will be rejected) password_secret: "../etc/passwd" # Path traversal password_secret: "/absolute/path" # Absolute path password_secret: "secret.name" # Special characters ``` 2. **Path Normalization**: All paths are normalized and validated to stay within `/app/secrets` - Relative paths are resolved relative to secrets directory - Absolute paths are rejected for secrets - Paths containing `../` or `..\\` are blocked 3. **Security Event Logging**: Path traversal attempts are logged for monitoring: ```json { "level": "error", "kind": "security_event", "type": "path_traversal_attempt", "secret_name": "../etc/passwd", "reason": "path_outside_allowed_directory" } ``` #### SSH Key Path Protection SSH key paths include similar protections: 1. **Traversal Pattern Detection**: Paths containing `..` patterns are rejected ```yaml # ✅ Valid key paths key_path: "id_ed25519" key_path: "prod_key" key_path: "/app/keys/id_ed25519" # Absolute within keys_dir # ❌ Invalid (will be rejected) key_path: "../outside_key" # Path traversal key_path: "/etc/passwd" # Outside keys_dir key_path: "key/../../etc/passwd" # Encoded traversal ``` 2. **Absolute Path Validation**: Absolute paths must be within the configured `keys_dir` - Paths outside `/app/keys` (or custom keys_dir) are rejected - Prevents accessing keys from other locations 3. **Security Event Logging**: All path traversal attempts are logged with full context **Effect**: Prevents reading files outside intended directories, blocking common path traversal attack vectors. #### File Type Validation All resolved paths are validated to ensure they are regular files: 1. **Directory Rejection**: Paths pointing to directories are rejected ```yaml # ❌ Invalid (will be rejected) password_secret: "subdirectory" # Points to a directory, not a file ``` 2. **Symlink Rejection**: Symbolic links are rejected for security ```yaml # ❌ Invalid (will be rejected) password_secret: "symlink_secret" # Points to a symlink, not a regular file ``` **Why reject symlinks?** Symlinks can be manipulated to point outside the allowed directory or to sensitive files, creating security risks. 3. **Regular File Requirement**: Only regular files within the allowed directory are accepted - Non-existent files are allowed for SSH keys (validated when used) - Secrets must exist as regular files (immediate read required) - All paths must stay within their designated directories 4. **Security Event Logging**: File validation failures are logged: ```json { "level": "error", "kind": "security_event", "type": "file_validation_failed", "file_path": "/app/secrets/subdirectory", "reason": "path_is_directory" } ``` **Effect**: Prevents accessing directories or symlinks that could lead to security vulnerabilities or unauthorized access. #### YAML File Size Limits All YAML configuration files are validated for size before loading to prevent resource exhaustion attacks: 1. **Size Limit**: Maximum file size of 10MB per YAML file - Applies to: `servers.yml`, `credentials.yml`, `policy.yml` - Prevents resource exhaustion via oversized configuration files - Files exceeding limit are rejected with security event logging 2. **Size Validation**: File size is checked using `os.path.getsize()` before parsing - Prevents loading files into memory if they exceed the limit - Returns empty dictionary on size limit violation - No YAML parsing performed if file is too large 3. **Security Event Logging**: Size limit violations are logged: ```json { "level": "error", "kind": "security_event", "type": "file_size_limit_exceeded", "path": "/app/config/servers.yml", "file_size": 10485761, "max_size": 10485760, "reason": "yaml_file_too_large" } ``` 4. **Normal Operation**: Files at or below the 10MB limit load normally - 10MB is sufficient for typical configuration files - Large configuration files (multiple thousands of hosts) are supported - Prevents abuse while allowing legitimate use cases **Effect**: Prevents resource exhaustion attacks via oversized YAML files that could consume excessive memory or processing time. #### Input Validation for User-Controlled Parameters All user-controlled parameters are validated before processing to prevent injection attacks and resource exhaustion: 1. **Alias Validation**: - Length limit: 100 characters - Allowed characters: alphanumeric, dash (`-`), underscore (`_`), dot (`.`) - Rejects empty values - Applied to: `ssh_describe_host`, `ssh_plan`, `ssh_run`, `ssh_run_async` 2. **Command Validation**: - Length limit: 10,000 characters - Rejects null bytes (`\x00`) - common injection vector - Rejects control characters (except newline `\n`, tab `\t`, carriage return `\r`) - Allows legitimate multi-line commands - Applied to: `ssh_plan`, `ssh_run`, `ssh_run_on_tag`, `ssh_run_async` 3. **Tag Validation**: - Length limit: 50 characters - Allowed characters: alphanumeric, dash (`-`), underscore (`_`), dot (`.`) - Rejects empty values - Applied to: `ssh_run_on_tag` 4. **Task ID Validation**: - Length limit: 200 characters - Allowed characters: alphanumeric, colon (`:`), dash (`-`), underscore (`_`) - Format validation: expected pattern `alias:hash:timestamp` - Applied to: `ssh_cancel`, `ssh_get_task_status`, `ssh_get_task_result`, `ssh_get_task_output`, `ssh_cancel_async_task` 5. **Security Event Logging**: Invalid input attempts are logged: ```json { "level": "error", "msg": "security_event", "type": "null_byte_injection_attempt", "field": "command" } ``` **Effect**: Prevents injection attacks (null bytes, control characters) and resource exhaustion (length limits) via malformed user inputs. #### Input Length Limits for Configuration Parameters String parameters in configuration files have length limits to prevent resource exhaustion: 1. **Secret Names** (`credentials.yml`): - Maximum length: 100 characters - Validated in `_resolve_secret()` function - Rejects names exceeding limit with security event logging 2. **SSH Key Paths** (`credentials.yml`): - Maximum length: 500 characters - Validated in `_resolve_key_path()` function - Rejects paths exceeding limit with security event logging 3. **MCP Tool Parameters** (from PR6): - **Alias**: Maximum 100 characters - **Command**: Maximum 10,000 characters - **Tag**: Maximum 50 characters - **Task ID**: Maximum 200 characters 4. **Length Validation Order**: Length validation occurs before other validations (character validation, path traversal checks) to prevent processing of oversized inputs. 5. **Security Event Logging**: Length limit violations are logged: ```json { "level": "error", "kind": "security_event", "type": "input_length_limit_exceeded", "field": "secret_name", "length": 150, "max_length": 100, "reason": "secret_name_too_long" } ``` **Effect**: Prevents resource exhaustion attacks via oversized string inputs in configuration files and user-controlled parameters. #### DNS Rate Limiting DNS resolution is rate-limited and cached to prevent DNS-based DoS attacks: 1. **Rate Limiting**: - Maximum 10 DNS resolutions per second per hostname - Per-hostname rate limiting (different hostnames have separate limits) - Time-window based (sliding 1-second window) - Exceeding limit returns empty list (no IPs resolved) 2. **Result Caching**: - DNS results cached for 60 seconds (TTL) - Cached results returned immediately without DNS lookup - Reduces load on DNS servers - Caches both successful and failed resolutions (prevents repeated lookups for invalid hostnames) 3. **Timeout Protection**: - DNS resolution timeout: 5 seconds - Prevents hanging on slow or unresponsive DNS servers - Failed resolutions return empty list 4. **Rate Limit Logging**: Rate limit violations are logged: #### Command Denial Bypass Prevention Command denial logic has been enhanced to prevent bypass attempts via obfuscation: 1. **Command Normalization**: - Removes single and double quotes from commands - Removes escaped characters (e.g., `\ ` becomes space) - Normalizes whitespace (collapses multiple spaces/tabs to single space) - Applied before checking against `deny_substrings` list 2. **Dual Checking**: - Checks original command string (maintains existing behavior) - Checks normalized command string (catches obfuscated bypass attempts) - Both checks must pass for command to be allowed 3. **Token-Based Matching**: - Splits normalized command into tokens - Checks for exact token matches against deny patterns - Prevents partial bypasses (e.g., `rm -rf /var` vs `rm -rf /`) 4. **Bypass Attempt Detection**: Commands that would bypass original checking but are caught by normalization are logged: ```json { "level": "error", "msg": "security_event", "type": "dns_rate_limit_exceeded", "hostname": "malicious-host.example.com", "max_per_second": 10 } ``` 5. **Thread Safety**: Rate limiter and cache are thread-safe for concurrent access. **Effect**: Prevents DNS-based DoS attacks by limiting resolution frequency and caching results, reducing load on DNS infrastructure. "type": "command_bypass_attempt", "alias": "web1", "original_command": "'rm -rf /'", "normalized_command": "rm -rf /", "blocked_pattern": "rm -rf /" } ``` 5. **Supported Bypass Techniques Prevented**: - Quote obfuscation: `'rm -rf /'`, `"rm -rf /"` - Escaped characters: `rm\ -rf\ /` - Whitespace variations: `rm -rf /`, `rm\t-rf\t/` - Mixed techniques: `echo "rm\\ -rf\\ /"` 6. **Limitations**: - Perfect prevention would require full command parsing - Complex obfuscation (base64 encoding, variable substitution) may still bypass - Focus is on common bypass techniques, not all possible obfuscation methods **Effect**: Significantly reduces risk of command denial bypasses through common obfuscation techniques (quotes, escaping, whitespace). ### SSH Key Management **Best Practices:** 1. **Use Ed25519 Keys**: Prefer `ed25519` over RSA for modern security ```bash ssh-keygen -t ed25519 -f ~/.ssh/mcp_orchestrator -C "mcp-ssh-orchestrator" ``` 2. **Key Permissions**: Set private keys to read-only for owner ```bash chmod 0400 /path/to/keys/id_ed25519 ``` 3. **Separate Keys**: Use dedicated keys for the orchestrator (not your personal keys) 4. **Key Rotation**: Rotate keys periodically (quarterly recommended) 5. **Passphrase Protection**: Use passphrases for private keys ```yaml # credentials.yml entries: - name: prod_admin username: ubuntu key_path: id_ed25519 key_passphrase_secret: prod_key_passphrase ``` ### Password Authentication **Avoid password authentication when possible.** If required: 1. **Use Secrets**: Never hardcode passwords in YAML ```yaml # credentials.yml entries: - name: legacy_system username: admin password_secret: legacy_password # Resolved from secret ``` 2. **Secret Resolution Order**: - Environment variable: `MCP_SSH_SECRET_<NAME>` (uppercase) - Docker secret file: `/app/secrets/<name>` 3. **Strong Passwords**: Minimum 16 characters, high entropy 4. **Limited Scope**: Use password auth only for hosts that don't support keys ### Secrets Storage **Docker Secrets (Recommended for Production):** ```bash # Create secret echo "my-passphrase" | docker secret create ssh_key_passphrase - # Use in Docker Compose services: mcp-ssh: secrets: - ssh_key_passphrase secrets: ssh_key_passphrase: external: true ``` **Environment Variables (Development Only):** ```bash docker run -i --rm \ -e MCP_SSH_SECRET_ADMIN_PASSWORD="dev-password" \ ghcr.io/samerfarida/mcp-ssh-orchestrator:0.1.0 ``` **File-Based Secrets (Simple Deployments):** ```bash # Create secrets directory mkdir -p ~/mcp-ssh/secrets chmod 0700 ~/mcp-ssh/secrets # Add secret echo "passphrase" > ~/mcp-ssh/secrets/key_passphrase chmod 0400 ~/mcp-ssh/secrets/key_passphrase ``` ## Network Security ### Host Key Verification **Always enable in production:** ```yaml # policy.yml limits: require_known_host: true network: require_known_host: true # Overrides limits setting ``` **Populate known_hosts:** ```bash # Scan host keys ssh-keyscan -H 10.0.0.11 >> ~/mcp-ssh/keys/known_hosts ssh-keyscan -H 10.0.0.21 >> ~/mcp-ssh/keys/known_hosts # Or copy from existing cp ~/.ssh/known_hosts ~/mcp-ssh/keys/ ``` **Effect**: Prevents MITM attacks by verifying host identity before connection. ### IP Allowlisting **Restrict SSH targets to known networks:** ```yaml # policy.yml network: # Allow only RFC1918 private networks allow_cidrs: - "10.0.0.0/8" - "172.16.0.0/12" - "192.168.0.0/16" # Block specific IPs block_ips: - "10.10.10.10" ``` **Two-Stage Verification:** 1. **Pre-Connect**: DNS resolution must match allowlist 2. **Post-Connect**: Actual peer IP must match allowlist **Effect**: Prevents connections to unexpected IPs, mitigates DNS poisoning. ### Egress Controls **Block lateral movement tools by default:** ```yaml # policy.yml limits: deny_substrings: - "ssh " - "scp " - "rsync -e ssh" - "curl " - "wget " - "nc " - "nmap " ``` **Effect**: Prevents compromised hosts from initiating outbound connections. ## Policy Enforcement ### Deny-by-Default **Start with no access, add only what's needed:** ```yaml # policy.yml rules: # Allow safe read-only commands - action: "allow" aliases: ["*"] tags: [] commands: - "uname*" - "uptime*" - "df -h*" # All other commands denied by default ``` ### Glob Pattern Safety **Use specific patterns to avoid overly broad matches:** ```yaml # BAD: Too permissive commands: - "*" # GOOD: Specific commands commands: - "systemctl status nginx" - "systemctl restart nginx" ``` ### Environment Separation **Use tags to separate production from non-production:** ```yaml # Policy for production rules: - action: "allow" aliases: [] tags: ["production"] commands: - "uptime*" - "df -h*" - action: "deny" aliases: [] tags: ["production"] commands: - "systemctl restart*" # Denied on prod # Policy for staging (more permissive) rules: - action: "allow" aliases: [] tags: ["staging"] commands: - "systemctl restart*" # Allowed on staging ``` ### Per-Host Overrides **Apply stricter limits to sensitive hosts:** ```yaml # policy.yml overrides: aliases: prod-db-1: max_seconds: 20 # Shorter timeout max_output_bytes: 262144 # Smaller output cap require_known_host: true # Strict host key check ``` ## Execution Limits ### Timeout Enforcement **Prevent runaway commands:** ```yaml # policy.yml limits: max_seconds: 60 # Global default overrides: tags: production: max_seconds: 30 # Stricter for prod longrun: max_seconds: 300 # Exception for maintenance ``` ### Output Size Caps **Prevent memory exhaustion:** ```yaml # policy.yml limits: max_output_bytes: 1048576 # 1 MiB default overrides: aliases: log-server: max_output_bytes: 10485760 # 10 MiB for log hosts ``` ### Deny Dangerous Substrings **Block destructive commands:** ```yaml # policy.yml limits: deny_substrings: - "rm -rf /" - ":(){ :|:& };:" # Fork bomb - "mkfs " - "dd if=/dev/zero" - "shutdown" - "reboot" - "userdel" - "passwd" ``` ## Container Security ### Non-Root Execution Container runs as UID 10001 (non-root): ```dockerfile RUN useradd -u 10001 -m appuser USER appuser ``` **Effect**: Limits damage if container is compromised. ### Read-Only Mounts **Mount configuration and keys as read-only:** ```bash docker run -i --rm \ -v ~/mcp-ssh/config:/app/config:ro \ -v ~/mcp-ssh/keys:/app/keys:ro \ ghcr.io/samerfarida/mcp-ssh-orchestrator:0.1.0 ``` **Effect**: Prevents accidental or malicious modification of config/keys. ### Minimal Base Image Uses `python:3.13-slim`: - Smaller attack surface - Fewer packages to patch - Reduced image size ### Health Checks ```dockerfile HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ CMD python -c "import mcp_ssh" || exit 1 ``` **Effect**: Early detection of container failures. ## Audit & Monitoring ### Security Audit Logging Security-relevant events are logged to stderr in structured JSON format via `_log_security_event()`: **Security Audit Log Format:** ```json { "level": "error", "kind": "security_audit", "type": "security_event", "event_type": "path_traversal_attempt", "ts": 1762112167.394149, "timestamp": "2025-11-02T14:36:07-0500", "attempted_path": "../etc/passwd", "resolved_path": "/app/secrets/../etc/passwd", "reason": "path_outside_allowed_directory", "base_dir": "/app/secrets" } ``` **Event Types Logged:** - `path_traversal_attempt`: Path traversal detected in secret/key resolution - `file_validation_failed`: Directory/symlink/non-file paths rejected - `file_size_limit_exceeded`: Oversized YAML files rejected - `input_length_limit_exceeded`: Oversized string inputs rejected - `invalid_secret_name`: Invalid characters in secret names - `dns_rate_limit_exceeded`: DNS resolution rate limit violations - `command_bypass_attempt`: Command denial bypass attempts detected **Audit Log Fields:** - `ts`: Unix timestamp (float) - `timestamp`: ISO 8601 formatted timestamp (string) - `attempted_path`: Original input that triggered the event - `resolved_path`: Resolved/absolute path (if applicable) - `reason`: Human-readable reason for the security event - `additional_data`: Event-specific context (sizes, limits, patterns, etc.) **Security Monitoring:** All security audit events are written to stderr for log aggregation and SIEM integration. ### Command Execution Audit Logging All operations logged to stderr as JSON: ```json { "type": "audit", "ts": 1729512345.67, "alias": "prod-web-1", "hash": "a1b2c3d4e5f6", "exit_code": 0, "duration_ms": 123, "bytes_out": 45, "bytes_err": 0, "cancelled": false, "timeout": false, "target_ip": "10.0.0.11" } ``` **Fields:** - `type`: Event type (audit, policy_decision, progress) - `ts`: Unix timestamp - `alias`: Target host - `hash`: Command hash (first 12 chars of SHA256) - `exit_code`: Command exit status - `duration_ms`: Execution time - `bytes_out/bytes_err`: Output size - `cancelled/timeout`: Termination reason - `target_ip`: Actual peer IP connected ### Log Collection **Docker Compose with logging driver:** ```yaml services: mcp-ssh: logging: driver: "json-file" options: max-size: "10m" max-file: "3" ``` **Forward to SIEM:** ```bash docker logs -f mcp-ssh-orchestrator 2>&1 | \ jq -r 'select(.type == "audit") | @json' | \ curl -X POST https://siem.example.com/ingest -d @- ``` ### Monitoring Recommendations **Alert on:** - Policy denials (especially repeated) - Timeouts - Non-zero exit codes on critical hosts - Unexpected target IPs - High-privilege commands (sudo, systemctl) **Metrics to track:** - Commands per hour - Success rate by host - Average duration - Policy violation rate ## Incident Response ### Compromised Key 1. **Immediately revoke** the key on all target hosts: ```bash # Remove from authorized_keys on each host ssh user@host "sed -i '/mcp-orchestrator/d' ~/.ssh/authorized_keys" ``` 2. **Rotate keys**: ```bash # Generate new key ssh-keygen -t ed25519 -f new_key # Deploy to hosts for host in $(cat hosts.txt); do ssh-copy-id -i new_key.pub user@$host done ``` 3. **Update configuration**: ```yaml # credentials.yml entries: - name: prod_admin key_path: new_key # Updated ``` 4. **Review audit logs** for unauthorized usage ### Policy Bypass Detected 1. **Stop the orchestrator** immediately 2. **Review policy rules** for gaps 3. **Check audit logs** for pattern 4. **Patch policy**: ```yaml # Add deny rule for bypass technique limits: deny_substrings: - "<bypass pattern>" ``` 5. **Reload config**: `ssh_reload_config` ### Unauthorized Access 1. **Check who accessed**: ```bash # Audit log analysis jq -r 'select(.alias == "compromised-host") | [.ts, .hash, .target_ip] | @tsv' < audit.log ``` 2. **Identify command hashes** executed 3. **Correlate with target host logs** 4. **Contain affected hosts** 5. **Rotate credentials** ## Security Framework Alignment ### OWASP LLM Top 10 Coverage **LLM07: Insecure Plugin Design** ✅ - Policy-based command validation prevents unauthorized execution - Input sanitization and dangerous command blocking - Access control for AI plugin operations **LLM08: Excessive Agency** ✅ - Role-based restrictions via host tags - Deny-by-default security model - Command pattern matching limits autonomous actions **LLM01: Prompt Injection Mitigation** - SSH command validation prevents injection attacks - Network egress controls block unauthorized connections - DNS verification prevents DNS rebinding attacks ### MITRE ATT&CK Alignment - **T1071**: Application Layer Protocol (SSH monitoring) - **T1071.004**: DNS (DNS resolution verification) - **T1659**: Content Injection (policy-based command filtering) ### Security Features Supporting Compliance This tool provides security controls that can support organizations seeking compliance with frameworks like SOC 2, ISO 27001, PCI-DSS, and HIPAA: **Access Control & Audit** - Policy enforcement provides least-privilege access - JSON logs provide non-repudiation and complete audit trails - All administrative access logged **Encryption & Network Security** - SSH provides transport encryption - Ed25519/RSA keys for strong cryptography - IP allowlists enforce network segmentation **Secrets Management** - Docker secrets or environment variables (not hardcoded) - No persistent credential storage **Note**: Compliance is ultimately the responsibility of the deploying organization. This tool provides security features that can support compliance efforts but is not itself certified to these standards. ## Security Checklist **Before Production Deployment:** - [ ] Use Ed25519 or RSA 4096-bit keys - [ ] Set private key permissions to 0400 - [ ] Enable `require_known_host: true` - [ ] Populate known_hosts file - [ ] Configure IP allowlists (allow_cidrs) - [ ] Enable deny_substrings for dangerous commands - [ ] Use deny-by-default policy model - [ ] Mount config and keys as read-only (:ro) - [ ] Use Docker secrets or env vars (not hardcoded passwords) - [ ] Set up audit log collection - [ ] Configure alerting on policy violations - [ ] Document incident response procedures - [ ] Test policy with `ssh_plan` before `ssh_run` - [ ] Separate production from non-production (tags) - [ ] Set appropriate timeouts and output limits - [ ] Review and minimize container privileges - [ ] Enable Docker health checks - [ ] Use specific command patterns (avoid overly broad globs) - [ ] Rotate keys quarterly - [ ] Keep Docker image updated ## Reporting Security Issues **Do not open public issues for security vulnerabilities.** Email: security@example.com (replace with your contact) Include: - Description of vulnerability - Steps to reproduce - Potential impact - Suggested fix (optional) We aim to respond within 48 hours and patch critical issues within 7 days.