# OpsPilot MCP — AI DevOps Agent with a Custom MCP Server

After building DeployMate AI as a RAG diagnostic assistant, this project
explores the next layer: how agents actually *act* on real systems, not just
reason about text. OpsPilot is a real MCP (Model Context Protocol) server
exposing Kubernetes-style diagnostic tools to an LLM agent, plus a
from-scratch agent loop that connects to it exactly the way Claude Code
connects to any MCP server under the hood.

No real Kubernetes cluster is needed — `fake_cluster.json` mocks cluster
state, so the whole thing runs anywhere in a couple of minutes, and every
piece of the MCP/agent mechanics has been verified against the real,
official `mcp` Python SDK.

---

## 1. System architecture

```
+-----------------------------------------------------------------------+
|                              YOU                                      |
|              "Why is payment-api failing?"                            |
+----------------------------------+------------------------------------+
                                   |
                                   v
+-----------------------------------------------------------------------+
|                       agent.py (MCP CLIENT)                           |
|                                                                         |
|   1. Connects to server.py over stdio (subprocess)                    |
|   2. Discovers available tools                                        |
|   3. Sends question + tool list to Claude                             |
|   4. Executes whatever tool Claude asks for                           |
|   5. Feeds the result back, repeats until Claude gives a final answer |
+----------------------------------+------------------------------------+
                                   |  MCP protocol (stdio, JSON-RPC under the hood)
                                   v
+-----------------------------------------------------------------------+
|                       server.py (MCP SERVER)                          |
|                                                                         |
|   Read-only tools:                Gated (destructive) tool:           |
|   - get_pod_status()              - restart_pod()                     |
|   - get_logs()                       -> returns PENDING, does not     |
|   - analyze_deployment()                execute                       |
|   - run_health_check()            - approve_action()                  |
|   - create_incident_report()         -> executes a pending action     |
+----------------------------------+------------------------------------+
                                   |  reads
                                   v
                        +----------------------+
                        |  fake_cluster.json    |   (stand-in for a real
                        |  (mock state)         |    Kubernetes API / kubectl)
                        +----------------------+

  Claude is the actual reasoning engine -- the MCP server has no
  intelligence of its own, it only executes exactly what it's told.
```

**The one-sentence version:** `agent.py` is a thin loop that lets Claude
reason; `server.py` is a thin layer that lets Claude *act*; neither one is
smart on its own — the intelligence is entirely in Claude's reasoning, and
MCP is just the standardized wire format connecting the two.

---

## 2. The agent loop, step by step, for one real query

This is the exact Reason -> Act -> Observe pattern behind every coding
agent, made fully visible in this project's terminal output:

```
 TURN 1
 -----------------------------------------------------------------
 User: "Why is payment-api failing?"
 Claude reasons: "I don't know yet -- I should check pod status first."
 Claude calls:    get_pod_status(namespace="production")
 -----------------------------------------------------------------
                              |
              agent.py executes the real tool call against
              server.py, gets back:
              payment-api -> CrashLoopBackOff, 14 restarts
                              |
                              v
 TURN 2
 -----------------------------------------------------------------
 Claude reasons: "CrashLoopBackOff -- I need the logs to see why
                  it's crashing."
 Claude calls:    get_logs(pod="payment-api")
 -----------------------------------------------------------------
                              |
              Result: "...OutOfMemoryError... exit code 137"
                              |
                              v
 TURN 3
 -----------------------------------------------------------------
 Claude reasons: "OOM -- let me check the configured memory limit
                  to confirm it's undersized."
 Claude calls:    analyze_deployment(deployment="payment-api")
 -----------------------------------------------------------------
                              |
              Result: memory_limit: "256Mi"
                              |
                              v
 TURN 4 (final)
 -----------------------------------------------------------------
 Claude has enough evidence. No more tool calls.
 Returns a final diagnosis: OOMKilled due to a 256Mi memory limit
 that's too low for the workload -- recommends raising it to
 512Mi, citing exit code 137 and the configured limit as evidence.
 -----------------------------------------------------------------
```

Notice Claude decided the *order* of investigation itself — nothing in the
code hardcodes "always check logs after pod status." That sequencing is the
actual reasoning the model is doing, turn by turn, driven by what each tool
result reveals.

---

## 3. The approval gate — a state diagram

This is the security-relevant part of the design, and worth being able to
draw from memory:

```
                    restart_pod(pod, reason) called
                                 |
                                 v
                    +------------------------+
                    |    PENDING_APPROVAL     |
                    |  (nothing has happened   |
                    |   to the real system     |
                    |   yet -- this is just a  |
                    |   recorded request)      |
                    +------------+-------------+
                                 |
              approve_action(id) called with the SAME id
                                 |
                  +--------------+---------------+
                  |                              |
            valid, unused id              invalid or already-used id
                  |                              |
                  v                              v
          +---------------+              +----------------+
          |    EXECUTED     |              |     ERROR       |
          | (action runs,   |              | (fails clearly,  |
          |  removed from   |              |  server does NOT |
          |  pending list -- |              |  crash, action   |
          |  can't be        |              |  is NOT          |
          |  approved again) |              |  re-executed)    |
          +---------------+              +----------------+
```

The key property, verified by `test_scenarios.py`: an action can transition
from `PENDING_APPROVAL` to `EXECUTED` exactly once. A second approval
attempt on the same id — or an approval attempt on a made-up id — fails
gracefully rather than silently re-running a destructive action or crashing
the server.

---

## 4. What's in this repo

| File | Role |
|---|---|
| `server.py` | The MCP server — exposes 7 tools, reads `fake_cluster.json` |
| `agent.py` | The MCP client + agent loop — connects to the server, talks to Claude, executes tool calls |
| `fake_cluster.json` | Mock cluster state standing in for a real Kubernetes API |
| `test_scenarios.py` | 8 scenarios validating server behavior directly, no API key needed |
| `run_all_scenarios.py` | Runs the real Claude-powered agent across multiple demo questions |

## 5. Test scenarios — what each one proves

| # | Scenario | What it proves |
|---|---|---|
| 1 | CrashLoopBackOff / OOM (`payment-api`) | The full diagnostic chain returns correct, consistent data across all four read tools |
| 2 | ImagePullBackOff (`notification-worker`) | The system handles a *different* failure mode, not just one hardcoded case |
| 3 | Healthy service (`auth-service`) | The system correctly reports "nothing's wrong" — a good agent shouldn't invent a problem just because it was asked to investigate |
| 4 | Querying a pod/deployment that doesn't exist | Errors are handled gracefully — a clear message, not a crash |
| 5 | Querying an empty/unknown namespace | Returns an empty result, not an error, for a namespace with no matches |
| 6 | Approval gate happy path | `restart_pod` → pending → `approve_action` → executed, confirming the gate works end to end |
| 7 | Approving an unknown action id | Fails gracefully instead of crashing the server |
| 8 | Re-approving an already-executed action | An action can't be executed twice — proves the gate isn't just cosmetic |

Run all 8 free, with no API key needed:
```bash
python3 test_scenarios.py
```

## Run the real agent

```bash
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
export ANTHROPIC_API_KEY=your_key_here

python3 agent.py "Why is payment-api failing?"
python3 agent.py "Is anything wrong with notification-worker?"
python3 agent.py "Is auth-service healthy right now?"

# or run several demo questions in one go:
python3 run_all_scenarios.py
```

Watch the terminal — it prints tool discovery, every tool call with its
exact arguments, every result returned, and the final diagnosis. Nothing is
hidden; this is the whole loop made visible.

## 6. Design decisions

**Why MCP instead of just writing custom Python functions Claude could call
directly?** MCP standardizes tool discovery and execution the same way
regardless of what's behind it — a Kubernetes cluster today, a GitHub API or
a database tomorrow — without rewriting the agent loop each time. It's also
exactly the mechanism real tools like Claude Code use, so building on it
directly is the most honest way to demonstrate this skill.

**Why mock the cluster instead of connecting to a real one?** The goal was
to demonstrate the *agent and protocol mechanics* correctly and testably, not
to stand up Kubernetes infrastructure. Swapping `fake_cluster.json` reads for
real Kubernetes Python client calls inside each tool function is a contained
change — the MCP layer and the agent loop wouldn't need to change at all.

**Why is `restart_pod` gated behind human approval, but the diagnostic tools
aren't?** This is a deliberate least-privilege design. The agent can gather
unlimited information because reads are safe — but write/destructive actions
against production are exactly the category the JD's "security and
compliance" concerns are about. Rather than trusting the agent's judgment
about *when* it's safe to restart something, the system makes it structurally
impossible to do so without an explicit human step — the same "instruction
vs. enforcement" principle behind hooks in Claude Code, applied to an MCP
tool boundary instead.

**Why does `create_incident_report` exist as a tool at all, if it's just
formatting text?** It forces the agent to commit to a structured, explicit
root cause and evidence, in a fixed format, rather than describing a vague
diagnosis in prose. That structure is what makes it easy to spot when the
agent's reasoning is weak (e.g. it can't fill in a specific evidence field)
versus genuinely well-grounded.

## What's mocked vs. real

- **Real:** the MCP server/client protocol itself, tool discovery, tool
  execution, the Claude tool-calling agent loop, the approval-gate logic —
  all verified directly against the real `mcp` Python SDK
- **Mocked:** the underlying data source (`fake_cluster.json` instead of a
  live Kubernetes API or GitHub API)

## Status

A working prototype. All 8 test scenarios pass against the real MCP server
mechanics with no API key required. The live agent loop (`agent.py`,
`run_all_scenarios.py`) requires an `ANTHROPIC_API_KEY` and has been run
successfully end to end.