┌───────────────────────────── CLUSTER HEALTHY ──────────────────────────────┐
│                                                                            │
│   LLM ──► kubectl MCP ──► kube-apiserver ──► etcd            ✓ works    │
│                                                                            │
└────────────────────────────────────────────────────────────────────────────┘

┌────────────────────────── CONTROL PLANE IS DOWN ───────────────────────────┐
│                                                                            │
│   LLM ──► kubectl MCP ──► kube-apiserver  ✗                  ✗ dead     │
│                              nothing left to talk to                       │
│                                                                            │
│   LLM ──► k8s-dr-mcp  ──► SSH ──► the node itself            ✓ works    │
│                                      systemctl · etcdctl ·                 │
│                                      kubeadm · crictl · journalctl         │
│                                                                            │
└────────────────────────────────────────────────────────────────────────────┘

pip install -e .

cp config.example.yaml            cluster.yaml   # generic kubeadm cluster
cp cluster.stacked.example.yaml   cluster.yaml   # Kubespray: stacked etcd + Calico
cp cluster.host-etcd.example.yaml cluster.yaml   # Kubespray: host etcd + Cilium

cluster:
  name: my-cluster
  etcd_deployment: stacked        # stacked | external | host
  container_runtime: containerd   # containerd | crio
  cni_plugin: calico

  ssh_defaults:
    user: root
    key_path: ~/.ssh/id_ed25519
    sudo: true

  # Optional — only if the nodes live on a private network
  # bastion:
  #   host: bastion.example.com
  #   user: ubuntu
  #   key_path: ~/.ssh/bastion_key

  control_plane:
    - { name: cp-1, host: 10.0.0.11, etcd_member_name: cp-1 }
    - { name: cp-2, host: 10.0.0.12, etcd_member_name: cp-2 }
    - { name: cp-3, host: 10.0.0.13, etcd_member_name: cp-3 }

  workers:
    - { name: worker-1, host: 10.0.0.31 }
    - { name: worker-2, host: 10.0.0.32 }

// .mcp.json in your project root
{
  "mcpServers": {
    "k8s-dr": {
      "type": "stdio",
      "command": "k8s-dr-mcp",
      "env": {
        "K8S_DR_MODE": "readonly",
        "K8S_DR_CLUSTER_CONFIG": "./cluster.yaml"
      }
    }
  }
}

You     The cluster is down — kubectl just hangs.

Claude  → dr_cluster_overview()
        → dr_check_etcd_health()
        → dr_check_etcd_alarms()

        etcd on all three nodes has a NOSPACE alarm — the database hit the
        2 GiB default quota. Fix is defrag + disarm. That's destructive and
        the server is read-only. Want me to walk you through, or restart in
        readwrite mode?

You     Restart in readwrite. Go ahead.

Claude  → dr_etcd_defrag(node="cp-1", confirm=True)        ↳ backup first
        → dr_etcd_alarm_disarm(node="cp-1", confirm=True)
        → …cp-2, cp-3…
        → dr_check_etcd_health()

        All three members healthy, quorum restored, API server answering.

Claude calls: dr_etcd_defrag(node="cp-1", confirm=True)
       │
       ▼
┌────────────────────┐
│  GATE 1 · MODE     │  K8S_DR_MODE == "readwrite"?
│                    │
│  readonly  ───────►│──► BLOCKED  "Set K8S_DR_MODE=readwrite"
│  readwrite ───────►│──► pass
└─────────┬──────────┘
          ▼
┌────────────────────┐
│  GATE 2 · CONFIRM  │  confirm == True?
│                    │
│  confirm=False ───►│──► PREVIEW  what it would do + pre-flight results,
│                    │             nothing executed
│  confirm=True  ───►│──► pass
└─────────┬──────────┘
          ▼
┌────────────────────┐
│  GATE 3 · PREFLIGHT│  ✓ SSH reachable      ✓ required binary present
│                    │  ✓ service in the     ✓ target dir writable
│                    │    expected state     ✓ enough disk space
│  any check failed ►│──► BLOCKED  with the specific failure
│  all passed    ───►│──► pass
└─────────┬──────────┘
          ▼
┌────────────────────┐
│  AUTO-BACKUP       │  etcd data dir · PKI · static pod manifests
└─────────┬──────────┘  (path returned in the result, and logged)
          ▼
┌────────────────────┐
│  EXECUTE           │  run it over SSH
└─────────┬──────────┘
          ▼
┌────────────────────┐
│  AUDIT             │  one JSON line → stderr and/or K8S_DR_AUDIT_LOG
└────────────────────┘

{"timestamp":"2026-07-28T09:14:22.881Z","tool":"dr_etcd_defrag","node":"cp-1",
 "mode":"readwrite","result":"success","confirm":true,
 "backup_path":"/var/lib/etcd-backup/etcd-20260728-091422.tar.gz","duration_ms":4130}

cluster:
  name: my-cluster
  etcd_deployment: stacked        # stacked | external | host
  container_runtime: containerd   # containerd | crio
  cni_plugin: calico
  cert_management: kubeadm        # kubeadm | script

  ssh_defaults:
    user: root
    key_path: ~/.ssh/id_ed25519
    port: 22
    sudo: true
    connect_timeout: 30
    command_timeout: 120

  bastion:                        # omit entirely for direct access
    host: ""
    user: ""
    key_path: ""
    port: 22

  control_plane:                  # required, at least one
    - name: cp-1
      host: 10.0.0.11
      ip: 10.0.0.11
      etcd_member_name: cp-1
      ssh_user: ubuntu            # optional per-node override
      ssh_key_path: ~/.ssh/cp1    # optional per-node override

  etcd_nodes: []                  # only when etcd_deployment: external
  workers: []                     # optional, for kubelet/runtime/network tools

  paths:                          # override only what differs
    etcd_data_dir:      /var/lib/etcd
    etcd_backup_dir:    /var/lib/etcd-backup
    k8s_manifests:      /etc/kubernetes/manifests
    k8s_pki:            /etc/kubernetes/pki
    k8s_pki_etcd:       /etc/kubernetes/pki/etcd
    kubelet_config:     /var/lib/kubelet/config.yaml
    kubelet_kubeconfig: /etc/kubernetes/kubelet.conf
    kubelet_flags:      /var/lib/kubelet/kubeadm-flags.env
    kubelet_service:    /etc/systemd/system/kubelet.service
    kubelet_dropin:     /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
    containerd_config:  /etc/containerd/config.toml
    crio_config:        /etc/crio/crio.conf
    cni_bin:            /opt/cni/bin
    cni_config:         /etc/cni/net.d
    etcd_cert_dir:      /etc/kubernetes/pki/etcd
    etcd_ca_cert:       ca.crt
    etcd_client_cert:   server.crt     # supports a {node} placeholder
    etcd_client_key:    server.key     # supports a {node} placeholder
    etcd_env_file:      ""             # Kubespray: /etc/etcd.env

cluster:
  etcd_deployment: host           # systemctl, not a static pod
  cert_management: script         # not kubeadm

  paths:
    k8s_pki:           /etc/kubernetes/ssl
    k8s_pki_etcd:      /etc/ssl/etcd/ssl
    etcd_cert_dir:     /etc/ssl/etcd/ssl
    etcd_ca_cert:      ca.pem
    etcd_client_cert:  "admin-{node}.pem"     # {node} → the node's name
    etcd_client_key:   "admin-{node}-key.pem"
    etcd_env_file:     /etc/etcd.env
    kubelet_config:    /etc/kubernetes/kubelet-config.yaml
    kubelet_flags:     /etc/kubernetes/kubelet.env

{
  "mcpServers": {
    "k8s-dr": {
      "type": "stdio",
      "command": "k8s-dr-mcp",
      "env": {
        "K8S_DR_MODE": "readonly",
        "K8S_DR_CLUSTER_CONFIG": "/absolute/path/to/cluster.yaml"
      }
    }
  }
}

{
  "mcpServers": {
    "k8s-dr": {
      "command": "python",
      "args": ["-m", "k8s_dr_mcp"],
      "env": {
        "K8S_DR_MODE": "readonly",
        "K8S_DR_CLUSTER_CONFIG": "/absolute/path/to/cluster.yaml"
      }
    }
  }
}

{
  "mcpServers": {
    "k8s-dr": {
      "command": "python",
      "args": ["-m", "k8s_dr_mcp"],
      "env": {
        "K8S_DR_MODE": "readonly",
        "K8S_DR_CLUSTER_CONFIG": "/absolute/path/to/cluster.yaml"
      }
    }
  }
}

        ┌──────────┐
        │  Claude  │  sees every tool from both servers
        └────┬─────┘
             │
       ┌─────┴───────────────────┐
       ▼                         ▼
┌──────────────┐        ┌──────────────────┐
│  k8s-dr-mcp  │        │  kubectl MCP     │
│  63 DR tools │        │  server          │
│              │        │                  │
│  SSH → nodes │        │  K8s API         │
│              │        │                  │
│  when the    │        │  when the        │
│  API is DOWN │        │  cluster is OK   │
└──────────────┘        └──────────────────┘

"env": { "K8S_DR_MODE": "readwrite" }

docker build -t k8s-dr-mcp .

docker run --rm -i \
  -v ./cluster.yaml:/config/cluster.yaml:ro \
  -v ./ssh:/config/ssh:ro \
  -e K8S_DR_MODE=readonly \
  k8s-dr-mcp

{
  "mcpServers": {
    "k8s-dr": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run", "--rm", "-i",
        "-v", "./cluster.yaml:/config/cluster.yaml:ro",
        "-v", "./ssh:/config/ssh:ro",
        "-e", "K8S_DR_MODE=readonly",
        "k8s-dr-mcp:latest"
      ]
    }
  }
}

K8S_DR_TOOL_GROUPS=diagnostics,etcd,certificates

  all groups · 63 tools              selected · 27 tools
┌───────────────────────┐          ┌───────────────────────┐
│ diagnostics      10   │          │ diagnostics      10   │ ✓ always on
│ etcd             10   │          │ etcd             10   │ ✓
│ certificates      7   │   ────►  │ certificates      7   │ ✓
│ control-plane     7   │          │                       │
│ kubelet           8   │          │  ~57% fewer tool      │
│ runtime           6   │          │  definitions in the   │
│ network           9   │          │  context window       │
│ recovery          6   │          └───────────────────────┘
└───────────────────────┘

                        ┌──────────────────────────┐
                        │   MCP client             │
                        │   Claude Code · Desktop  │
                        │   Cursor · Windsurf      │
                        └────────────┬─────────────┘
                                     │ JSON-RPC over stdio
                        ┌────────────▼─────────────┐
                        │   k8s-dr-mcp  (FastMCP)  │
                        │                          │
                        │   tool router            │ ← K8S_DR_TOOL_GROUPS
                        │   safety gates           │ ← mode · confirm · preflight
                        │   auto-backup            │
                        │   audit logger           │ → JSON, stderr or file
                        └────────────┬─────────────┘
                                     │ AsyncSSH connection pool
                                     │ cached · auto-reconnect · parallel
                        ┌────────────▼─────────────┐
                        │   bastion / jump host    │  optional
                        └────────────┬─────────────┘
                                     │ tunneled SSH
              ┌──────────────────────┼──────────────────────┐
              │                      │                      │
     ┌────────▼────────┐    ┌────────▼────────┐    ┌────────▼────────┐
     │  cp-1           │    │  cp-2           │    │  cp-3           │
     │  etcd           │    │  etcd           │    │  etcd           │
     │  kube-apiserver │    │  kube-apiserver │    │  kube-apiserver │
     │  scheduler      │    │  scheduler      │    │  scheduler      │
     │  controller-mgr │    │  controller-mgr │    │  controller-mgr │
     │  kubelet        │    │  kubelet        │    │  kubelet        │
     └─────────────────┘    └─────────────────┘    └─────────────────┘

       systemctl · etcdctl · kubeadm · crictl · journalctl · openssl · ss · ip

src/k8s_dr_mcp/
├── server.py          FastMCP instance, SSH pool lifespan
├── config.py          env settings + cluster YAML loading
├── models/            pydantic models — cluster topology, tool results
├── ssh/               AsyncSSH connection pool, command executor
├── safety/            gates · preflight checks · auto-backup · audit log
└── tools/             the 8 tool groups

pip install -e ".[dev]"

pytest tests/ -v            # test suite
ruff check src/ tests/      # lint
mypy src/                   # type check (advisory — see below)