# KubeHealer

An AI agent that heals broken Kubernetes pods — fronted by a **crash-proof MCP
server**, made durable by **Temporal**.

<img src="docs/images/dashboard.png" alt="KubeHealer Mission Control dashboard" width="840">

---

## The idea worth understanding

A normal MCP server keeps its work *inside the process*. Kill it mid-task and
the state, the LLM's diagnoses and all in-flight progress die with it.

KubeHealer's MCP server keeps the work *inside Temporal* and is just a thin
pointer to it:

```text
heal_cluster (MCP tool)   ──►  start HealerWorkflow   (id = kubehealer-heal-<ns>)
get_healing_status        ──►  HealerWorkflow.get_state       (query)
approve_fix / reject_fix  ──►  HealerWorkflow.approve/reject  (signals)
```

The workflow ID is **deterministic per namespace**, so an MCP server that
crashes and restarts re-attaches to the same running heal by recomputing the
ID — no lost steps, and no fix applied twice.

Kill the MCP server mid-heal and the Temporal workflow keeps running. Restart
it and the client reconnects and finishes. That contrast is the whole point,
and [`CRASHPROOF_MCP.md`](CRASHPROOF_MCP.md) walks through it with naive-vs-durable
code side by side.

<img src="docs/images/temporal-ui.png" alt="Temporal Web UI: HealerWorkflow still running after the MCP server was killed" width="840">

*The heal workflow stays `Running` in Temporal even while the MCP server is dead.*

## Quick start

Prerequisites: Python 3.11+, Docker, Kind, the Temporal CLI, an Anthropic API key.

```bash
./setup.sh                    # kind cluster + broken demo pods
pip install -r requirements.txt
cp .env.example .env          # add your ANTHROPIC_API_KEY
```

Then one process per terminal — `make mcp` is the one you deliberately break:

```bash
make temporal      # durable backend + Web UI  (:8233)
make worker        # runs the workflows + activities
make mcp           # the MCP server            ← Ctrl-C this to break the demo
make dashboard     # Mission Control           (:8090)
```

`make help` lists every target; [`RUN.md`](RUN.md) has the full recipes, ports and
the break-and-recover walkthrough.

`.env` is gitignored — keep it that way, the API key belongs nowhere near a commit.

## Three ways to drive it

| Mode | Command | Path |
|------|---------|------|
| GUI — Mission Control | `make dashboard` | MCP → Temporal |
| CLI — agent brain | `make agent` | MCP → Temporal |
| Headless — one shot | `make auto` | Temporal directly |

Plus `make cli` for plain chat straight to Temporal with no MCP in the way, and
`make mcp-naive` for the non-durable "before" server that makes the contrast
obvious.

## The demo

1. **Break Pods** — three services go red
2. **Heal (with approval)** — the AI diagnoses each one; each pod heals the moment you approve it
3. **Break the MCP server** — the MCP plane flatlines, Temporal stays live and keeps healing; restart and it reconnects

| Broken service | Problem | AI fix |
|----------------|---------|--------|
| `storefront` | image `nginx:latestt` (typo) | patch → `nginx:latest` |
| `checkout-api` | 10Mi limit + stress → OOMKilled | raise the memory limit |
| `catalog-cache` | image `redis:latst` (typo) | patch → `redis:latest` |

## Architecture

```text
  CLI agent ┐                    ┌─────────────────────┐
  Dashboard ┼─ MCP (HTTP) ─────► │  MCP server         │  thin & disposable
  (clients) ┘   reconnects       │  (FastMCP)          │  — kill it anytime
                                 └──────────┬──────────┘
                              start · signal · query
                                            ▼
                       ┌──────────────────────────────────────┐
                       │  TEMPORAL · HealerWorkflow           │  durable plane —
                       │  scan → diagnose → approve → fix     │  state persisted
                       │  (each step a retryable activity)    │  at every step
                       └──────────────────┬───────────────────┘
                                runs on the Worker
                                          ▼
                                   ┌────────────┐
                                   │ Kubernetes │  (local kind cluster)
                                   └────────────┘
```

The dashboard also reads Temporal and Kubernetes **directly**, which is why the
Temporal plane and the pod cards keep advancing on screen even after the MCP
plane dies.

## Layout

```
mcp_server/
  server.py            FastMCP server — durable, delegates to Temporal
  naive_server.py      The non-durable contrast
  temporal_client.py   Deterministic workflow IDs, start/signal/query
workflows/
  healer_workflow.py       scan → diagnose → approve → fix
  heal_pod_workflow.py     per-pod child workflow
  conversation_workflow.py interactive agent loop
activities/
  llm_activities.py    Claude calls
  k8s_activities.py    Cluster reads and remediation
  chat_activities.py   Conversation plumbing
agent/brain.py         CLI agent — the MCP host
dashboard/             FastAPI gateway, hub, supervisor, sources
ui/                    React + Vite Mission Control frontend
chaos/                 Deliberately broken manifests
tests/                 pytest suite (workflows, agent, dashboard, e2e)
docs/images/           Screenshots used above
```

## Tests

```bash
make test                                    # pytest suite
pip install playwright && playwright install chromium   # optional, for the browser e2e
```

`tests/test_dashboard_e2e.py` skips itself if Playwright is not installed.

## Stack

Temporal (durable workflows) · Claude Sonnet 4 (diagnosis + agent) ·
FastMCP with SEP-1686 Tasks · FastAPI + React/Vite dashboard ·
Kubernetes / kind · Python 3.11+

## Notes

This applies real remediation to a real cluster — patching images and raising
limits without asking, in `make auto` mode. Point it at the kind cluster
`setup.sh` builds, not at anything you care about.

---

## Credits & license

KubeHealer originates from
[temporal-community/kubehealer](https://github.com/temporal-community/kubehealer),
MIT licensed — see [`LICENSE`](LICENSE). This copy tracks upstream `main`. The
original README is preserved as [`ORIGINAL-README.md`](ORIGINAL-README.md), and
[`SPEAKER_NOTES.md`](SPEAKER_NOTES.md) holds the demo walkthrough notes. My
changes and notes are my own.