# MCP Event-Driven DevOps Engine

A local infrastructure monitor that watches host resource usage and service health, streams
live telemetry to a React dashboard over SSE, and dispatches a local LLM (via Ollama) as an
autonomous remediation agent when it detects degraded services — with an MCP server exposing
the same monitoring/remediation tools to any MCP-compatible client (Claude Desktop, etc.).

## Problem

Local dev environments (Docker containers, Postgres, Redis, a dev API) drift out of a healthy
state silently — a container dies, a port stops responding, cache grows unbounded — and you
find out only when something downstream breaks. This project is a self-contained proof of
concept for closing that loop: detect degradation, hand it to an LLM agent with a constrained
tool surface, let it decide and execute a safe fix, and make the whole cycle observable.

## Architecture

```
┌─────────────────┐   poll every 4s    ┌──────────────────────┐
│  Host / Docker   │ ─────────────────▶ │   mcp-server (Node)  │
│  (CPU, RAM,      │                    │  - Express API       │
│   ports, docker) │                    │  - SSE broadcaster   │
└─────────────────┘                    │  - SQLite history     │
                                        │  - MCP tool server    │
                                        └─────────┬────────────┘
                                                   │ spawn (on degraded state)
                                                   ▼
                                        ┌──────────────────────┐
                                        │   agent.ts            │
                                        │  Ollama (qwen2.5-     │
                                        │  coder:7b) tool-call  │
                                        │  against real         │
                                        │  telemetry snapshot   │
                                        └─────────┬────────────┘
                                                   │ POST /api/agent-remediate
                                                   ▼
                                        ┌──────────────────────┐
                                        │  react-dashboard      │
                                        │  Vite + React +       │
                                        │  Tailwind + Recharts  │
                                        │  (live via SSE)       │
                                        └──────────────────────┘
```

**mcp-server** (`mcp-server/`) — Node/TypeScript/Express.
- Polls host CPU/RAM (`os.cpus()`), local port health (`net.Socket` connect checks on 5173/5432),
  and Docker status (`docker ps`) every 4 seconds.
- Persists metrics history and logs to SQLite (`telemetry.db`), keeping a sliding window (last
  50 log rows, last 50 metric snapshots).
- Broadcasts every tick to connected dashboards via Server-Sent Events (`/events`).
- Exposes an MCP server over stdio (`@modelcontextprotocol/sdk`) with three tools:
  `get_system_status`, `read_environment_logs`, `execute_environment_fix` — so any MCP client,
  not just this dashboard, can query or remediate the same environment.
- On detecting a degraded service (with a 30s cooldown to prevent cascading agent spawns), it
  spawns `agent.ts` as a child process, passing the **real live metrics snapshot** as an argument.

**agent.ts** (`mcp-server/src/agent.ts`) — the autonomous remediation step.
- Calls Ollama (`qwen2.5-coder:7b`) with a proper tool schema (`execute_environment_fix`) and
  reads back structured `tool_calls` from the response — not string-matching on free text.
- The prompt is built from the actual telemetry snapshot passed in (which services are
  degraded, current CPU/RAM/cache), not a hardcoded scripted alert.
- On a valid tool call, POSTs the chosen action back to the server's `/api/agent-remediate`
  endpoint, which executes it (e.g. `docker start postgres-dev`), and optionally forwards a
  remediation event to an n8n webhook for external automation.

**react-dashboard** (`react-dashboard/`) — Vite + React 19 + TypeScript + Tailwind + Recharts.
- Live CPU/RAM/cache tiles, a rolling telemetry area chart, per-service health tiles
  (api/database/docker_containers), a diagnostic log stream, and a manual "Force Manual Audit"
  trigger — all driven by the SSE stream with an initial REST fetch on load.
- Client-side anomaly heuristic (CPU up 25%+ over the last 4 snapshots) surfaces a predictive
  warning independently of the backend agent.

## A real finding from the eval harness

First run of `npm run eval:full` against `qwen2.5-coder:7b` (Ollama 0.32.5) scored 6/20 — every
case that should have produced a tool call instead produced none. The raw model output showed
the model reasoning correctly almost every time (e.g. `{"name": "execute_environment_fix",
"arguments": {"action": "restart_postgres"}}`) but emitting it as plain text in
`message.content` instead of Ollama's structured `tool_calls` field, which `decide()` was only
reading from the structured field. Confirmed the model itself wasn't the problem
(`ollama show qwen2.5-coder:7b` lists `tools` as a supported capability) before changing
anything.

Fix: `agentCore.decide()` now falls back to strict JSON-schema validation against the model's
raw text only when the structured field comes back empty — not the substring-matching approach
the original `agent.ts` used. One case (elevated cache, no degraded service) surfaced a genuine
hallucination — the model invented a tool name (`report_nominal_status`) that isn't in the
schema — which the fallback correctly treats as "no valid action" rather than accepting it.

After that fix, back-to-back runs of the same 20 cases scored differently (16/20, then 15/20)
with no code changes between them — Ollama's default sampling isn't deterministic, so a single
eval run's score wasn't trustworthy on its own. Pinned `temperature: 0`.

With temperature pinned, the score became reproducible — and revealed something a varying score
had been masking: every "all healthy, no action needed" case deterministically produced
`clear_cache` anyway, regardless of the actual cache number, while every real degraded-service
case was correct. The prompt had been asking the model to judge whether a cache number was
"far above normal" — a numeric threshold decision an LLM shouldn't be trusted to make reliably
when code can make it deterministically instead. Fix: `describeSituation()` now computes whether
cache is elevated (`CACHE_ELEVATED_THRESHOLD_MB = 100`) in code and tells the model an
unambiguous conclusion — "cache is elevated, call clear_cache" or "cache is nominal, do NOT call
any tool" — rather than a number and a vague instruction to reason about it.

After all five fixes (structured-field fallback, multi-shape JSON parsing, markdown-fence
stripping, pinned temperature, deterministic cache-threshold check), `npm run eval:full` scores
**20/20** against `qwen2.5-coder:7b`. 14 of those 20 still go through the fallback text parser —
this model reliably reasons correctly but has not been observed emitting Ollama's structured
`tool_calls` field even once across ~80 calls made while building this harness, so the fallback
path is load-bearing, not a rare edge case.

## Tradeoffs / design decisions

- **SQLite over a real time-series DB**: sufficient at this scale (single host, 50-row sliding
  window), avoids an extra service dependency for a local tool.
- **Polling (4s) over OS-level event hooks**: simpler, portable across platforms; costs
  responsiveness for very short-lived failures between ticks.
- **Cooldown-gated agent spawn (30s) instead of a queue**: prevents cascading agent loops when
  multiple services degrade at once, at the cost of possibly missing a fix window if a new
  issue appears mid-cooldown.
- **Local LLM (Ollama) over hosted API**: zero marginal cost and no data leaving the host, at
  the cost of weaker reasoning than a frontier hosted model — mitigated by keeping the tool
  surface small and explicit rather than relying on open-ended reasoning.
- **Separate child process per agent run** rather than an in-process call: isolates a slow or
  hung model call from the main event loop and dashboard responsiveness.

## Known limitations

- Remediation actions (`restart_redis`, `restart_postgres`, `clear_cache`) are a fixed,
  small action set — this is intentionally scoped as a proof of concept, not a general-purpose
  ops agent.
- No test suite yet; the mcp-server `test` script is a placeholder.
- n8n webhook forwarding degrades silently if n8n isn't running locally — this is expected in a
  standalone demo, but worth knowing before assuming remediation events are always externally
  visible.

## Evals

`mcp-server/src/evals/` scores the remediation agent's decisions against 20 synthetic
telemetry scenarios (`cases.ts`) — single degraded services, multiple simultaneous
degradations, healthy-but-high-resource states, and a no-data edge case.

Two stages, run via `npm run eval` (or `npm test`) inside `mcp-server/`:

1. **Prompt-construction check** (always runs, no LLM call, no Ollama required) — verifies
   `describeSituation()` actually surfaces the facts the model needs (which service is
   degraded, current CPU/RAM/cache) for every case. Deterministic, safe for CI.
2. **Full LLM-graded run** (`npm run eval:full`) — calls the real model via
   `agentCore.decide()` with `execute: false` (scores the decision, fires no docker
   restarts or webhooks) and checks whether the returned tool call matches the expected
   action. Requires a local Ollama instance running `qwen2.5-coder:7b`.

Note on ground truth: the degraded-service cases have an unambiguous correct action. The
cache-threshold cases (e.g. "150MB, no degraded service, should call `clear_cache`") encode
a judgment call rather than a hard rule, since the system prompt intentionally leaves the
threshold fuzzy ("elevated," "far above normal") rather than hardcoding a number — worth
knowing before treating a miss on those specific cases as a regression.

## Running it

Requires [Ollama](https://ollama.com) running locally with `qwen2.5-coder:7b` pulled, and
Docker running if you want the docker-status check and postgres remediation to do anything real.

```bash
# terminal 1 — MCP/API server
cd mcp-server
npm install
npm start

# terminal 2 — dashboard
cd react-dashboard
npm install
npm run dev
```

Dashboard: http://localhost:5173. API/SSE: http://localhost:3001.