KubeHealer

An AI agent that heals broken Kubernetes pods — fronted by a crash-proof MCP server, made durable by Temporal.

The idea worth understanding

A normal MCP server keeps its work inside the process. Kill it mid-task and the state, the LLM's diagnoses and all in-flight progress die with it.

KubeHealer's MCP server keeps the work inside Temporal and is just a thin pointer to it:

heal_cluster (MCP tool)   ──►  start HealerWorkflow   (id = kubehealer-heal-<ns>)
get_healing_status        ──►  HealerWorkflow.get_state       (query)
approve_fix / reject_fix  ──►  HealerWorkflow.approve/reject  (signals)

The workflow ID is deterministic per namespace, so an MCP server that crashes and restarts re-attaches to the same running heal by recomputing the ID — no lost steps, and no fix applied twice.

Kill the MCP server mid-heal and the Temporal workflow keeps running. Restart it and the client reconnects and finishes. That contrast is the whole point, and CRASHPROOF_MCP.md walks through it with naive-vs-durable code side by side.

The heal workflow stays Running in Temporal even while the MCP server is dead.

Quick start

Prerequisites: Python 3.11+, Docker, Kind, the Temporal CLI, an Anthropic API key.

./setup.sh                    # kind cluster + broken demo pods
pip install -r requirements.txt
cp .env.example .env          # add your ANTHROPIC_API_KEY

Then one process per terminal — make mcp is the one you deliberately break:

make temporal      # durable backend + Web UI  (:8233)
make worker        # runs the workflows + activities
make mcp           # the MCP server            ← Ctrl-C this to break the demo
make dashboard     # Mission Control           (:8090)

make help lists every target; RUN.md has the full recipes, ports and the break-and-recover walkthrough.

.env is gitignored — keep it that way, the API key belongs nowhere near a commit.

Three ways to drive it

Mode	Command	Path
GUI — Mission Control	`make dashboard`	MCP → Temporal
CLI — agent brain	`make agent`	MCP → Temporal
Headless — one shot	`make auto`	Temporal directly

Plus make cli for plain chat straight to Temporal with no MCP in the way, and make mcp-naive for the non-durable "before" server that makes the contrast obvious.

The demo

Break Pods — three services go red
Heal (with approval) — the AI diagnoses each one; each pod heals the moment you approve it
Break the MCP server — the MCP plane flatlines, Temporal stays live and keeps healing; restart and it reconnects

Broken service	Problem	AI fix
`storefront`	image `nginx:latestt` (typo)	patch → `nginx:latest`
`checkout-api`	10Mi limit + stress → OOMKilled	raise the memory limit
`catalog-cache`	image `redis:latst` (typo)	patch → `redis:latest`

Architecture

  CLI agent ┐                    ┌─────────────────────┐
  Dashboard ┼─ MCP (HTTP) ─────► │  MCP server         │  thin & disposable
  (clients) ┘   reconnects       │  (FastMCP)          │  — kill it anytime
                                 └──────────┬──────────┘
                              start · signal · query
                                            ▼
                       ┌──────────────────────────────────────┐
                       │  TEMPORAL · HealerWorkflow           │  durable plane —
                       │  scan → diagnose → approve → fix     │  state persisted
                       │  (each step a retryable activity)    │  at every step
                       └──────────────────┬───────────────────┘
                                runs on the Worker
                                          ▼
                                   ┌────────────┐
                                   │ Kubernetes │  (local kind cluster)
                                   └────────────┘

The dashboard also reads Temporal and Kubernetes directly, which is why the Temporal plane and the pod cards keep advancing on screen even after the MCP plane dies.

Layout

mcp_server/
  server.py            FastMCP server — durable, delegates to Temporal
  naive_server.py      The non-durable contrast
  temporal_client.py   Deterministic workflow IDs, start/signal/query
workflows/
  healer_workflow.py       scan → diagnose → approve → fix
  heal_pod_workflow.py     per-pod child workflow
  conversation_workflow.py interactive agent loop
activities/
  llm_activities.py    Claude calls
  k8s_activities.py    Cluster reads and remediation
  chat_activities.py   Conversation plumbing
agent/brain.py         CLI agent — the MCP host
dashboard/             FastAPI gateway, hub, supervisor, sources
ui/                    React + Vite Mission Control frontend
chaos/                 Deliberately broken manifests
tests/                 pytest suite (workflows, agent, dashboard, e2e)
docs/images/           Screenshots used above

Tests

make test                                    # pytest suite
pip install playwright && playwright install chromium   # optional, for the browser e2e

tests/test_dashboard_e2e.py skips itself if Playwright is not installed.

Stack

Temporal (durable workflows) · Claude Sonnet 4 (diagnosis + agent) · FastMCP with SEP-1686 Tasks · FastAPI + React/Vite dashboard · Kubernetes / kind · Python 3.11+

Notes

This applies real remediation to a real cluster — patching images and raising limits without asking, in make auto mode. Point it at the kind cluster setup.sh builds, not at anything you care about.

Credits & license

KubeHealer originates from temporal-community/kubehealer, MIT licensed — see LICENSE. This copy tracks upstream main. The original README is preserved as ORIGINAL-README.md, and SPEAKER_NOTES.md holds the demo walkthrough notes. My changes and notes are my own.