k8s-dr-mcp
by kulibins
README.md
<div align="center">
# k8s-dr-mcp
# !!! This is MVP version only !!! Running on production on your own risk and responsibility! !!!
### Kubernetes disaster recovery for AI agents — over SSH, for when the API server is already dead.
[](https://www.python.org/)
[](https://github.com/jlowin/fastmcp)
[](#tool-reference)
[](#safety-model)
[](#license)
[](#status)
</div>
---
Every Kubernetes MCP server on the market talks to the Kubernetes API. That is exactly the thing
that is gone when you actually need help at 3am.
**k8s-dr-mcp gives an AI agent a way in through the back door: SSH to the nodes themselves.**
`systemctl`, `etcdctl`, `kubeadm`, `crictl`, `journalctl`, `openssl` — the tools you'd reach for by
hand, wrapped in 63 MCP tools with a three-gate safety model so the agent can't wreck a cluster
that's already having a bad day.
```
┌───────────────────────────── CLUSTER HEALTHY ──────────────────────────────┐
│ │
│ LLM ──► kubectl MCP ──► kube-apiserver ──► etcd ✓ works │
│ │
└────────────────────────────────────────────────────────────────────────────┘
┌────────────────────────── CONTROL PLANE IS DOWN ───────────────────────────┐
│ │
│ LLM ──► kubectl MCP ──► kube-apiserver ✗ ✗ dead │
│ nothing left to talk to │
│ │
│ LLM ──► k8s-dr-mcp ──► SSH ──► the node itself ✓ works │
│ systemctl · etcdctl · │
│ kubeadm · crictl · journalctl │
│ │
└────────────────────────────────────────────────────────────────────────────┘
```
Run it **alongside** a normal kubectl-based MCP server. The agent picks the right one on its own:
API for healthy clusters, SSH for broken ones.
---
## Contents
| | |
|---|---|
| [Quick start](#quick-start) · [Disaster playbook](#disaster-playbook) · [Tool reference](#tool-reference) | [Safety model](#safety-model) · [Configuration](#configuration) · [Kubespray](#kubespray-clusters) |
| [MCP client setup](#mcp-client-setup) · [Docker](#docker) · [Context budget](#context-budget) | [How it works](#how-it-works) · [Development](#development) · [Security](#security) |
---
## Highlights
- **63 tools, 8 groups** — diagnostics, etcd, certificates, control plane, kubelet, runtime, network, and orchestrated recovery workflows
- **Works when nothing else does** — no API server, no kubeconfig, no CNI required
- **Read-only by default** — 43 of the 63 tools can't change anything; the other 20 need two explicit unlocks
- **Three-gate safety** — mode gate → confirmation gate → automated pre-flight checks, then an auto-backup
- **Parallel by design** — AsyncSSH connection pool fans out across every control-plane node at once
- **Bastion-native** — tunnels into private node networks through a jump host
- **Audit trail** — one structured JSON line per operation, to stderr or a file
- **Kubespray-aware** — host-mode etcd, `.pem` cert layouts, `/etc/etcd.env`, script-based cert renewal
---
## Quick start
### 1. Install
```bash
pip install -e .
```
### 2. Describe your cluster
Copy the example that matches your topology:
```bash
cp config.example.yaml cluster.yaml # generic kubeadm cluster
cp cluster.stacked.example.yaml cluster.yaml # Kubespray: stacked etcd + Calico
cp cluster.host-etcd.example.yaml cluster.yaml # Kubespray: host etcd + Cilium
```
Minimum viable config — node names, addresses, and how to SSH in:
```yaml
cluster:
name: my-cluster
etcd_deployment: stacked # stacked | external | host
container_runtime: containerd # containerd | crio
cni_plugin: calico
ssh_defaults:
user: root
key_path: ~/.ssh/id_ed25519
sudo: true
# Optional — only if the nodes live on a private network
# bastion:
# host: bastion.example.com
# user: ubuntu
# key_path: ~/.ssh/bastion_key
control_plane:
- { name: cp-1, host: 10.0.0.11, etcd_member_name: cp-1 }
- { name: cp-2, host: 10.0.0.12, etcd_member_name: cp-2 }
- { name: cp-3, host: 10.0.0.13, etcd_member_name: cp-3 }
workers:
- { name: worker-1, host: 10.0.0.31 }
- { name: worker-2, host: 10.0.0.32 }
```
> `cluster.yaml`, `.env`, `.mcp.json` and everything under `ssh/` are gitignored — they hold your
> real hostnames, addresses and keys. Only the `*.example.*` files are meant to be committed.
### 3. Wire it into your agent
```jsonc
// .mcp.json in your project root
{
"mcpServers": {
"k8s-dr": {
"type": "stdio",
"command": "k8s-dr-mcp",
"env": {
"K8S_DR_MODE": "readonly",
"K8S_DR_CLUSTER_CONFIG": "./cluster.yaml"
}
}
}
}
```
Other clients (Claude Desktop, Cursor, Windsurf, VS Code, Docker) → [MCP client setup](#mcp-client-setup).
### 4. Ask
```
You The cluster is down — kubectl just hangs.
Claude → dr_cluster_overview()
→ dr_check_etcd_health()
→ dr_check_etcd_alarms()
etcd on all three nodes has a NOSPACE alarm — the database hit the
2 GiB default quota. Fix is defrag + disarm. That's destructive and
the server is read-only. Want me to walk you through, or restart in
readwrite mode?
You Restart in readwrite. Go ahead.
Claude → dr_etcd_defrag(node="cp-1", confirm=True) ↳ backup first
→ dr_etcd_alarm_disarm(node="cp-1", confirm=True)
→ …cp-2, cp-3…
→ dr_check_etcd_health()
All three members healthy, quorum restored, API server answering.
```
---
## Disaster playbook
What to say, and what the agent will reach for.
| Symptom | Say something like | Tools it uses |
|---|---|---|
| `x509: certificate has expired` | *"Certs look expired — check and renew across the control plane."* | `dr_certs_check_expiration` → `dr_certs_backup` → `dr_certs_renew_all` ⚠ → `dr_cp_restart_all` ⚠ |
| `etcdserver: mvcc: database space exceeded` | *"etcd is out of space."* | `dr_check_etcd_alarms` → `dr_etcd_defrag` ⚠ → `dr_etcd_alarm_disarm` ⚠ |
| 2 of 3 control-plane nodes gone | *"We lost quorum. One survivor left."* | `dr_etcd_member_list` → `dr_recover_quorum_loss` ⚠ (force-new-cluster on the survivor, then re-add members) |
| API server won't come back after a manifest edit | *"apiserver won't start — compare cp-1 against the others."* | `dr_cp_validate_manifests` → `dr_cp_diff_manifests` → `dr_cp_get_manifest` → `dr_cp_restart_component` ⚠ |
| Node stuck `NotReady` | *"worker-3 is NotReady, find out why."* | `dr_kubelet_status` → `dr_kubelet_logs` → `dr_runtime_status` → `dr_net_cni_config` |
| Pods stuck `ContainerCreating` | *"CNI looks broken on this node."* | `dr_net_cni_config` → `dr_net_cni_binaries` → `dr_net_cilium_status` → `dr_net_dns` |
| Restored from a snapshot, unsure what's next | *"Give me a recovery plan before we touch anything."* | `dr_generate_recovery_plan` (read-only — diagnoses every node and writes out the steps) |
| Total loss, need it back now | *"Diagnose and recover the whole cluster."* | `dr_recover_full_cluster` ⚠ (diagnose → backup → repair → verify) |
⚠ = requires `K8S_DR_MODE=readwrite` **and** `confirm=True`.
---
## Tool reference
**63 tools across 8 groups.** 43 are read-only. 20 are destructive (⚠) and pass all three safety
gates before running. Every tool is prefixed `dr_`.
| Group | Tools | Destructive | Purpose |
|---|--:|--:|---|
| [`diagnostics`](#diagnostics) | 10 | 0 | Health, logs, ports, disk — always loaded |
| [`etcd`](#etcd) | 10 | 7 | Snapshots, restore, members, defrag, alarms |
| [`certificates`](#certificates) | 7 | 2 | Expiry, inspection, renewal, PKI backup |
| [`control-plane`](#control-plane) | 7 | 2 | Static pod manifests, component restarts |
| [`kubelet`](#kubelet) | 8 | 3 | Kubelet state, config, systemd services |
| [`runtime`](#runtime) | 6 | 1 | containerd / CRI-O, containers, images |
| [`network`](#network) | 9 | 0 | CNI, interfaces, firewall, DNS, Cilium, Hubble |
| [`recovery`](#recovery) | 6 | 5 | Multi-step orchestrated recovery workflows |
<details id="diagnostics">
<summary><b>diagnostics</b> — 10 tools, all read-only</summary>
Always registered, even if you leave it out of `K8S_DR_TOOL_GROUPS`.
| Tool | What it does |
|---|---|
| `dr_cluster_overview` | Unified health snapshot of every control-plane node |
| `dr_check_node` | Detailed system health for one node |
| `dr_check_etcd_health` | etcd cluster health, member status, quorum |
| `dr_check_etcd_alarms` | Active etcd alarms (`NOSPACE`, `CORRUPT`) |
| `dr_check_api_health` | kube-apiserver health on a node |
| `dr_check_ports` | Whether critical Kubernetes ports are listening |
| `dr_check_services` | systemd unit status |
| `dr_get_logs` | journald logs for a specific unit |
| `dr_check_static_pods` | List and validate `/etc/kubernetes/manifests/` |
| `dr_check_disk_usage` | Disk usage on the paths that matter |
</details>
<details id="etcd">
<summary><b>etcd</b> — 10 tools, 7 destructive</summary>
| Tool | What it does |
|---|---|
| `dr_etcd_snapshot` | Take a snapshot and verify its integrity |
| `dr_etcd_snapshot_status` | Inspect an existing snapshot file |
| `dr_etcd_member_list` | List cluster members |
| `dr_etcd_restore` ⚠ | Restore from a snapshot via `etcdutl` |
| `dr_etcd_force_new_cluster` ⚠ | Bootstrap a single-member cluster from a survivor — total quorum loss |
| `dr_etcd_member_add` ⚠ | Add a member |
| `dr_etcd_member_remove` ⚠ | Remove a member |
| `dr_etcd_alarm_disarm` ⚠ | Disarm active alarms |
| `dr_etcd_defrag` ⚠ | Defragment storage to reclaim space |
| `dr_etcd_compact` ⚠ | Compact history to a revision |
</details>
<details id="certificates">
<summary><b>certificates</b> — 7 tools, 2 destructive</summary>
| Tool | What it does |
|---|---|
| `dr_certs_check_expiration` | Expiry dates for all Kubernetes certificates |
| `dr_certs_inspect` | `openssl` inspection of one certificate |
| `dr_certs_inspect_all` | Inspect everything under the PKI directories |
| `dr_certs_check_connectivity` | Verify the TLS chain to a service |
| `dr_certs_backup` | Tarball the whole PKI directory |
| `dr_certs_renew_all` ⚠ | `kubeadm certs renew all` |
| `dr_certs_renew_single` ⚠ | Renew one certificate |
</details>
<details id="control-plane">
<summary><b>control-plane</b> — 7 tools, 2 destructive</summary>
| Tool | What it does |
|---|---|
| `dr_cp_get_manifest` | Read a static pod manifest |
| `dr_cp_validate_manifests` | YAML-validate every manifest |
| `dr_cp_backup_manifests` | Back up the manifests directory |
| `dr_cp_check_images` | Are the referenced images present locally? |
| `dr_cp_diff_manifests` | Diff manifests between two nodes — finds the odd one out |
| `dr_cp_restart_component` ⚠ | Restart one component by cycling its manifest |
| `dr_cp_restart_all` ⚠ | Restart all components in dependency order |
</details>
<details id="kubelet">
<summary><b>kubelet</b> — 8 tools, 3 destructive</summary>
| Tool | What it does |
|---|---|
| `dr_kubelet_status` | systemd status for kubelet |
| `dr_kubelet_config` | Read the kubelet config file |
| `dr_kubelet_flags` | Read startup flags (`kubeadm-flags.env`) |
| `dr_kubelet_kubeconfig` | Read the kubelet kubeconfig, secrets masked |
| `dr_kubelet_logs` | kubelet journal logs |
| `dr_kubelet_restart` ⚠ | Restart kubelet |
| `dr_service_start` ⚠ | Start any systemd unit |
| `dr_service_stop` ⚠ | Stop any systemd unit |
</details>
<details id="runtime">
<summary><b>runtime</b> — 6 tools, 1 destructive</summary>
| Tool | What it does |
|---|---|
| `dr_runtime_status` | containerd / CRI-O service status |
| `dr_runtime_config` | Read the runtime config file |
| `dr_runtime_containers` | List containers via `crictl` |
| `dr_runtime_container_logs` | Logs from one container |
| `dr_runtime_images` | Cached images on the node |
| `dr_runtime_restart` ⚠ | Restart the runtime |
</details>
<details id="network">
<summary><b>network</b> — 9 tools, all read-only</summary>
| Tool | What it does |
|---|---|
| `dr_net_cni_config` | CNI configuration files |
| `dr_net_cni_binaries` | Installed CNI plugin binaries |
| `dr_net_interfaces` | Interfaces and routing table |
| `dr_net_connectivity` | TCP reachability between two nodes |
| `dr_net_firewall` | Firewall rules |
| `dr_net_dns` | DNS config and resolution |
| `dr_net_cilium_status` | Cilium agent health |
| `dr_net_cilium_connectivity` | Quick Cilium connectivity test |
| `dr_net_hubble_status` | Hubble relay health and recent flows |
</details>
<details id="recovery">
<summary><b>recovery</b> — 6 orchestrated workflows, 5 destructive</summary>
Each of these chains many low-level tools into one guarded, multi-step procedure.
| Tool | What it does |
|---|---|
| `dr_generate_recovery_plan` | Diagnose every node and produce a plan — **read-only, start here** |
| `dr_recover_expired_certs` ⚠ | Back up PKI → renew on all CP nodes in parallel → restart → verify |
| `dr_recover_etcd_single` ⚠ | Restore etcd from a snapshot on one node |
| `dr_recover_etcd_cluster` ⚠ | Restore etcd from a snapshot across all CP nodes |
| `dr_recover_quorum_loss` ⚠ | Rebuild the cluster from a surviving member |
| `dr_recover_full_cluster` ⚠ | Auto-diagnose, then run whichever recovery fits |
</details>
---
## Safety model
The server is **read-only until you say otherwise**, and even then every destructive tool has to
clear three gates and take a backup before it touches anything.
```
Claude calls: dr_etcd_defrag(node="cp-1", confirm=True)
│
▼
┌────────────────────┐
│ GATE 1 · MODE │ K8S_DR_MODE == "readwrite"?
│ │
│ readonly ───────►│──► BLOCKED "Set K8S_DR_MODE=readwrite"
│ readwrite ───────►│──► pass
└─────────┬──────────┘
▼
┌────────────────────┐
│ GATE 2 · CONFIRM │ confirm == True?
│ │
│ confirm=False ───►│──► PREVIEW what it would do + pre-flight results,
│ │ nothing executed
│ confirm=True ───►│──► pass
└─────────┬──────────┘
▼
┌────────────────────┐
│ GATE 3 · PREFLIGHT│ ✓ SSH reachable ✓ required binary present
│ │ ✓ service in the ✓ target dir writable
│ │ expected state ✓ enough disk space
│ any check failed ►│──► BLOCKED with the specific failure
│ all passed ───►│──► pass
└─────────┬──────────┘
▼
┌────────────────────┐
│ AUTO-BACKUP │ etcd data dir · PKI · static pod manifests
└─────────┬──────────┘ (path returned in the result, and logged)
▼
┌────────────────────┐
│ EXECUTE │ run it over SSH
└─────────┬──────────┘
▼
┌────────────────────┐
│ AUDIT │ one JSON line → stderr and/or K8S_DR_AUDIT_LOG
└────────────────────┘
```
**Gate 2 is the useful one in practice.** Calling a destructive tool with `confirm=False` is a
dry run: you get a description of the operation plus the pre-flight results, and nothing happens.
Agents can safely explore what a fix would involve before proposing it.
Audit entries look like this:
```json
{"timestamp":"2026-07-28T09:14:22.881Z","tool":"dr_etcd_defrag","node":"cp-1",
"mode":"readwrite","result":"success","confirm":true,
"backup_path":"/var/lib/etcd-backup/etcd-20260728-091422.tar.gz","duration_ms":4130}
```
---
## Configuration
### Cluster file
Full schema, with defaults shown. Only `name` and `control_plane` are required.
```yaml
cluster:
name: my-cluster
etcd_deployment: stacked # stacked | external | host
container_runtime: containerd # containerd | crio
cni_plugin: calico
cert_management: kubeadm # kubeadm | script
ssh_defaults:
user: root
key_path: ~/.ssh/id_ed25519
port: 22
sudo: true
connect_timeout: 30
command_timeout: 120
bastion: # omit entirely for direct access
host: ""
user: ""
key_path: ""
port: 22
control_plane: # required, at least one
- name: cp-1
host: 10.0.0.11
ip: 10.0.0.11
etcd_member_name: cp-1
ssh_user: ubuntu # optional per-node override
ssh_key_path: ~/.ssh/cp1 # optional per-node override
etcd_nodes: [] # only when etcd_deployment: external
workers: [] # optional, for kubelet/runtime/network tools
paths: # override only what differs
etcd_data_dir: /var/lib/etcd
etcd_backup_dir: /var/lib/etcd-backup
k8s_manifests: /etc/kubernetes/manifests
k8s_pki: /etc/kubernetes/pki
k8s_pki_etcd: /etc/kubernetes/pki/etcd
kubelet_config: /var/lib/kubelet/config.yaml
kubelet_kubeconfig: /etc/kubernetes/kubelet.conf
kubelet_flags: /var/lib/kubelet/kubeadm-flags.env
kubelet_service: /etc/systemd/system/kubelet.service
kubelet_dropin: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
containerd_config: /etc/containerd/config.toml
crio_config: /etc/crio/crio.conf
cni_bin: /opt/cni/bin
cni_config: /etc/cni/net.d
etcd_cert_dir: /etc/kubernetes/pki/etcd
etcd_ca_cert: ca.crt
etcd_client_cert: server.crt # supports a {node} placeholder
etcd_client_key: server.key # supports a {node} placeholder
etcd_env_file: "" # Kubespray: /etc/etcd.env
```
### Environment variables
All prefixed `K8S_DR_`. Also readable from a `.env` file in the working directory.
| Variable | Default | Description |
|---|---|---|
| `K8S_DR_MODE` | `readonly` | `readonly` or `readwrite` |
| `K8S_DR_CLUSTER_CONFIG` | `./cluster.yaml` | Path to the cluster file |
| `K8S_DR_TOOL_GROUPS` | all 8 groups | Comma-separated groups to load |
| `K8S_DR_LOG_LEVEL` | `INFO` | `DEBUG` · `INFO` · `WARNING` · `ERROR` |
| `K8S_DR_AUDIT_LOG` | *(stderr only)* | Also append audit JSON to this file |
| `K8S_DR_SSH_USER` | `root` | Default SSH user |
| `K8S_DR_SSH_KEY_PATH` | `~/.ssh/id_ed25519` | Default private key |
| `K8S_DR_SSH_PORT` | `22` | Default SSH port |
| `K8S_DR_SSH_TIMEOUT` | `30` | Connect timeout, seconds |
| `K8S_DR_SSH_KNOWN_HOSTS` | `~/.ssh/known_hosts` | `none` disables host key verification |
| `K8S_DR_BASTION_HOST` | *(none)* | Jump host address |
| `K8S_DR_BASTION_USER` | *(none)* | Jump host user |
| `K8S_DR_BASTION_KEY_PATH` | *(none)* | Jump host key |
| `K8S_DR_BASTION_PORT` | `22` | Jump host port |
Values in `cluster.yaml` win over the environment for anything both can express — the env vars are
there for container deployments where mounting one file is easier than templating two.
---
## Kubespray clusters
Kubespray doesn't lay things out the way `kubeadm` does: etcd runs as a **systemd unit** rather than
a static pod, certificates are `.pem` files in `/etc/ssl/etcd/ssl`, and renewal goes through a
script instead of `kubeadm certs renew`. Tell the server about it and every tool adapts:
```yaml
cluster:
etcd_deployment: host # systemctl, not a static pod
cert_management: script # not kubeadm
paths:
k8s_pki: /etc/kubernetes/ssl
k8s_pki_etcd: /etc/ssl/etcd/ssl
etcd_cert_dir: /etc/ssl/etcd/ssl
etcd_ca_cert: ca.pem
etcd_client_cert: "admin-{node}.pem" # {node} → the node's name
etcd_client_key: "admin-{node}-key.pem"
etcd_env_file: /etc/etcd.env
kubelet_config: /etc/kubernetes/kubelet-config.yaml
kubelet_flags: /etc/kubernetes/kubelet.env
```
Two ready-made starting points ship with the repo: `cluster.stacked.example.yaml` (stacked etcd +
Calico) and `cluster.host-etcd.example.yaml` (host etcd + Cilium).
---
## MCP client setup
<details>
<summary><b>Claude Code</b></summary>
`.mcp.json` in the project root, or `~/.claude.json` for every project:
```json
{
"mcpServers": {
"k8s-dr": {
"type": "stdio",
"command": "k8s-dr-mcp",
"env": {
"K8S_DR_MODE": "readonly",
"K8S_DR_CLUSTER_CONFIG": "/absolute/path/to/cluster.yaml"
}
}
}
}
```
Without installing the entry point, use `"command": "python", "args": ["-m", "k8s_dr_mcp"]`.
Verify with `claude mcp list`. A copyable starting point lives in `.mcp.json.example`, including a
second server for the healthy-cluster case.
</details>
<details>
<summary><b>Claude Desktop</b></summary>
macOS `~/Library/Application Support/Claude/claude_desktop_config.json`,
Windows `%APPDATA%\Claude\claude_desktop_config.json`:
```json
{
"mcpServers": {
"k8s-dr": {
"command": "python",
"args": ["-m", "k8s_dr_mcp"],
"env": {
"K8S_DR_MODE": "readonly",
"K8S_DR_CLUSTER_CONFIG": "/absolute/path/to/cluster.yaml"
}
}
}
}
```
</details>
<details>
<summary><b>Cursor · Windsurf · VS Code</b></summary>
Cursor — `.cursor/mcp.json`; Windsurf — `~/.codeium/windsurf/mcp_config.json`:
```json
{
"mcpServers": {
"k8s-dr": {
"command": "python",
"args": ["-m", "k8s_dr_mcp"],
"env": {
"K8S_DR_MODE": "readonly",
"K8S_DR_CLUSTER_CONFIG": "/absolute/path/to/cluster.yaml"
}
}
}
}
```
VS Code — `.vscode/mcp.json`, same shape but the top-level key is `"servers"` and each entry needs
`"type": "stdio"`.
</details>
<details>
<summary><b>Run both DR and kubectl MCP servers together</b> — recommended</summary>
The agent chooses based on what's actually broken: kubectl while the API answers, SSH once it
doesn't.
```
┌──────────┐
│ Claude │ sees every tool from both servers
└────┬─────┘
│
┌─────┴───────────────────┐
▼ ▼
┌──────────────┐ ┌──────────────────┐
│ k8s-dr-mcp │ │ kubectl MCP │
│ 63 DR tools │ │ server │
│ │ │ │
│ SSH → nodes │ │ K8s API │
│ │ │ │
│ when the │ │ when the │
│ API is DOWN │ │ cluster is OK │
└──────────────┘ └──────────────────┘
```
See `.mcp.json.example` for a working two-server config, including the SSH port-forward wrapper for
reaching a private API endpoint through a bastion.
</details>
### Enabling write operations
```json
"env": { "K8S_DR_MODE": "readwrite" }
```
That only opens **gate 1**. Every destructive tool still needs `confirm=True` and still has to pass
pre-flight checks. Running read-only day to day and restarting in readwrite when you've decided on a
fix is a reasonable habit.
---
## Docker
```bash
docker build -t k8s-dr-mcp .
```
```bash
docker run --rm -i \
-v ./cluster.yaml:/config/cluster.yaml:ro \
-v ./ssh:/config/ssh:ro \
-e K8S_DR_MODE=readonly \
k8s-dr-mcp
```
> **Note:** the image sets `K8S_DR_MODE=readwrite` in its own `ENV` so it works out of the box for
> recovery. Pass `-e K8S_DR_MODE=readonly` (as above) — or edit the `Dockerfile` — if you'd rather
> the container default to safe.
The image runs as a non-root user, expects the cluster file at `/config/cluster.yaml`, and reads SSH
keys from wherever your config points — `/config/ssh/<key>` if you mount the repo's `ssh/`
directory. From an MCP client:
```json
{
"mcpServers": {
"k8s-dr": {
"type": "stdio",
"command": "docker",
"args": [
"run", "--rm", "-i",
"-v", "./cluster.yaml:/config/cluster.yaml:ro",
"-v", "./ssh:/config/ssh:ro",
"-e", "K8S_DR_MODE=readonly",
"k8s-dr-mcp:latest"
]
}
}
}
```
---
## Context budget
63 tool definitions is a lot of context to hand an LLM before it has done anything. Load only the
groups the situation calls for:
```bash
K8S_DR_TOOL_GROUPS=diagnostics,etcd,certificates
```
```
all groups · 63 tools selected · 27 tools
┌───────────────────────┐ ┌───────────────────────┐
│ diagnostics 10 │ │ diagnostics 10 │ ✓ always on
│ etcd 10 │ │ etcd 10 │ ✓
│ certificates 7 │ ────► │ certificates 7 │ ✓
│ control-plane 7 │ │ │
│ kubelet 8 │ │ ~57% fewer tool │
│ runtime 6 │ │ definitions in the │
│ network 9 │ │ context window │
│ recovery 6 │ └───────────────────────┘
└───────────────────────┘
```
`diagnostics` is always registered whether you list it or not — you can't triage without it.
---
## How it works
```
┌──────────────────────────┐
│ MCP client │
│ Claude Code · Desktop │
│ Cursor · Windsurf │
└────────────┬─────────────┘
│ JSON-RPC over stdio
┌────────────▼─────────────┐
│ k8s-dr-mcp (FastMCP) │
│ │
│ tool router │ ← K8S_DR_TOOL_GROUPS
│ safety gates │ ← mode · confirm · preflight
│ auto-backup │
│ audit logger │ → JSON, stderr or file
└────────────┬─────────────┘
│ AsyncSSH connection pool
│ cached · auto-reconnect · parallel
┌────────────▼─────────────┐
│ bastion / jump host │ optional
└────────────┬─────────────┘
│ tunneled SSH
┌──────────────────────┼──────────────────────┐
│ │ │
┌────────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐
│ cp-1 │ │ cp-2 │ │ cp-3 │
│ etcd │ │ etcd │ │ etcd │
│ kube-apiserver │ │ kube-apiserver │ │ kube-apiserver │
│ scheduler │ │ scheduler │ │ scheduler │
│ controller-mgr │ │ controller-mgr │ │ controller-mgr │
│ kubelet │ │ kubelet │ │ kubelet │
└─────────────────┘ └─────────────────┘ └─────────────────┘
systemctl · etcdctl · kubeadm · crictl · journalctl · openssl · ss · ip
```
The SSH pool keeps one connection per node, opens the bastion tunnel once and reuses it, reconnects
on drop, and fans out across all control-plane nodes concurrently — so `dr_cluster_overview` on a
three-node cluster costs about as long as querying one node.
Everything the server does is an ordinary shell command over SSH. Nothing is installed on the nodes,
and nothing depends on the cluster being able to schedule a pod.
### Layout
```
src/k8s_dr_mcp/
├── server.py FastMCP instance, SSH pool lifespan
├── config.py env settings + cluster YAML loading
├── models/ pydantic models — cluster topology, tool results
├── ssh/ AsyncSSH connection pool, command executor
├── safety/ gates · preflight checks · auto-backup · audit log
└── tools/ the 8 tool groups
```
---
## Development
```bash
pip install -e ".[dev]"
pytest tests/ -v # test suite
ruff check src/ tests/ # lint
mypy src/ # type check (advisory — see below)
```
Adding a tool: write it in the right `tools/*.py` module, register it in that module's `register()`,
and if it changes anything on a node, route it through `run_safety_gates()` with a `confirm: bool`
parameter and a pre-flight check.
### What CI enforces
`.github/workflows/ci.yml` runs on every push and pull request:
| Job | Gate |
|---|---|
| `lint` | `ruff check` — **blocking** |
| `test` | `pytest` on Python 3.10 / 3.11 / 3.12 / 3.13 — **blocking** |
| `build` | wheel + sdist, `twine check` — **blocking** |
| `docker` | image builds *and* still defaults to `K8S_DR_MODE=readonly` — **blocking** |
| `secrets` | no private-key material, no tracked `.env` / `cluster.yaml` / `*.pem`, nothing in `ssh/` but its README — **blocking** |
| `types` | `mypy src/` — **advisory** |
Type-checking is advisory on purpose. FastMCP tools receive their context as `ctx: Context = None`
— the framework injects it at call time, but mypy reads the `None` default as an implicit Optional
and flags every such signature. Annotating `Context | None` just relocates the complaint to each
`ctx.` use site, so the honest options are a large signature rework or an advisory job. If you make
the type errors go away without weakening the checks, that's a very welcome PR.
### Tech
- **Python 3.10+**, asyncio throughout
- **[FastMCP](https://github.com/jlowin/fastmcp)** — MCP server framework
- **[AsyncSSH](https://asyncssh.readthedocs.io/)** — non-blocking SSH; parallel node fan-out without a thread pool
- **[Pydantic](https://docs.pydantic.dev/)** — config validation and structured tool results
---
## Security
This server executes privileged shell commands on your Kubernetes nodes, usually with `sudo`. Treat
it as the sensitive thing it is:
- **Keep it read-only** unless you are actively recovering something.
- **Use a dedicated SSH key** for the server rather than your personal one, and scope it as tightly
as your setup allows.
- **Never commit keys or real inventories.** `cluster.yaml`, `.env`, `.mcp.json` and `ssh/*` are
gitignored for exactly this reason — if one slips through, rotate the key; deleting the file isn't
enough.
- **Turn on host key verification in production.** `K8S_DR_SSH_KNOWN_HOSTS=none` disables it, which
is convenient for a lab and a bad idea anywhere else.
- **Point `K8S_DR_AUDIT_LOG` at a real file** when running against production, so there's a record
of what ran and when.
Found a vulnerability? Please report it privately rather than opening a public issue.
## Status
Alpha. The tooling is complete and covered by tests, but it has been exercised against a limited
set of cluster topologies (kubeadm and Kubespray, containerd, Calico and Cilium). Try it on a
throwaway cluster before you trust it with a real outage — ideally *rehearse* your recovery with it,
so the first time you use it in anger isn't the first time you use it.
Reports about other topologies — CRI-O, external etcd, flannel, k3s — are especially welcome.
## Contributing
Issues and pull requests are welcome. Useful contributions, roughly in order of value:
1. Support for cluster layouts that don't work yet
2. New recovery workflows for failure modes you've actually hit
3. Pre-flight checks that would have caught something the hard way
Please run `ruff` and the tests before opening a PR, and keep destructive tools behind the
three gates.
## License
MIT.
---
<div align="center">
<sub><b>Rehearse your disaster recovery before you need it.</b><br>
The best time to find out this works on your cluster is a Tuesday afternoon, not a Saturday night.</sub>
</div>
This server cannot be deployed
Maintenance
ActivitySlowing
ResponsivenessNo issues