MCP Event-Driven DevOps Engine

A local infrastructure monitor that watches host resource usage and service health, streams live telemetry to a React dashboard over SSE, and dispatches a local LLM (via Ollama) as an autonomous remediation agent when it detects degraded services — with an MCP server exposing the same monitoring/remediation tools to any MCP-compatible client (Claude Desktop, etc.).

Problem

Local dev environments (Docker containers, Postgres, Redis, a dev API) drift out of a healthy state silently — a container dies, a port stops responding, cache grows unbounded — and you find out only when something downstream breaks. This project is a self-contained proof of concept for closing that loop: detect degradation, hand it to an LLM agent with a constrained tool surface, let it decide and execute a safe fix, and make the whole cycle observable.

Related MCP server: WEATHGARDS

Architecture

┌─────────────────┐   poll every 4s    ┌──────────────────────┐
│  Host / Docker   │ ─────────────────▶ │   mcp-server (Node)  │
│  (CPU, RAM,      │                    │  - Express API       │
│   ports, docker) │                    │  - SSE broadcaster   │
└─────────────────┘                    │  - SQLite history     │
                                        │  - MCP tool server    │
                                        └─────────┬────────────┘
                                                   │ spawn (on degraded state)
                                                   ▼
                                        ┌──────────────────────┐
                                        │   agent.ts            │
                                        │  Ollama (qwen2.5-     │
                                        │  coder:7b) tool-call  │
                                        │  against real         │
                                        │  telemetry snapshot   │
                                        └─────────┬────────────┘
                                                   │ POST /api/agent-remediate
                                                   ▼
                                        ┌──────────────────────┐
                                        │  react-dashboard      │
                                        │  Vite + React +       │
                                        │  Tailwind + Recharts  │
                                        │  (live via SSE)       │
                                        └──────────────────────┘

mcp-server (mcp-server/) — Node/TypeScript/Express.

• Polls host CPU/RAM (os.cpus()), local port health (net.Socket connect checks on 5173/5432), and Docker status (docker ps) every 4 seconds.

• Persists metrics history and logs to SQLite (telemetry.db), keeping a sliding window (last 50 log rows, last 50 metric snapshots).

• Broadcasts every tick to connected dashboards via Server-Sent Events (/events).

• Exposes an MCP server over stdio (@modelcontextprotocol/sdk) with three tools: get_system_status, read_environment_logs, execute_environment_fix — so any MCP client, not just this dashboard, can query or remediate the same environment.

• On detecting a degraded service (with a 30s cooldown to prevent cascading agent spawns), it spawns agent.ts as a child process, passing the real live metrics snapshot as an argument.

agent.ts (mcp-server/src/agent.ts) — the autonomous remediation step.

• Calls Ollama (qwen2.5-coder:7b) with a proper tool schema (execute_environment_fix) and reads back structured tool_calls from the response — not string-matching on free text.

• The prompt is built from the actual telemetry snapshot passed in (which services are degraded, current CPU/RAM/cache), not a hardcoded scripted alert.

• On a valid tool call, POSTs the chosen action back to the server's /api/agent-remediate endpoint, which executes it (e.g. docker start postgres-dev), and optionally forwards a remediation event to an n8n webhook for external automation.

react-dashboard (react-dashboard/) — Vite + React 19 + TypeScript + Tailwind + Recharts.

• Live CPU/RAM/cache tiles, a rolling telemetry area chart, per-service health tiles (api/database/docker_containers), a diagnostic log stream, and a manual "Force Manual Audit" trigger — all driven by the SSE stream with an initial REST fetch on load.

• Client-side anomaly heuristic (CPU up 25%+ over the last 4 snapshots) surfaces a predictive warning independently of the backend agent.

A real finding from the eval harness

First run of npm run eval:full against qwen2.5-coder:7b (Ollama 0.32.5) scored 6/20 — every case that should have produced a tool call instead produced none. The raw model output showed the model reasoning correctly almost every time (e.g. {"name": "execute_environment_fix", "arguments": {"action": "restart_postgres"}}) but emitting it as plain text in message.content instead of Ollama's structured tool_calls field, which decide() was only reading from the structured field. Confirmed the model itself wasn't the problem (ollama show qwen2.5-coder:7b lists tools as a supported capability) before changing anything.

Fix: agentCore.decide() now falls back to strict JSON-schema validation against the model's raw text only when the structured field comes back empty — not the substring-matching approach the original agent.ts used. One case (elevated cache, no degraded service) surfaced a genuine hallucination — the model invented a tool name (report_nominal_status) that isn't in the schema — which the fallback correctly treats as "no valid action" rather than accepting it.

After that fix, back-to-back runs of the same 20 cases scored differently (16/20, then 15/20) with no code changes between them — Ollama's default sampling isn't deterministic, so a single eval run's score wasn't trustworthy on its own. Pinned temperature: 0.

With temperature pinned, the score became reproducible — and revealed something a varying score had been masking: every "all healthy, no action needed" case deterministically produced clear_cache anyway, regardless of the actual cache number, while every real degraded-service case was correct. The prompt had been asking the model to judge whether a cache number was "far above normal" — a numeric threshold decision an LLM shouldn't be trusted to make reliably when code can make it deterministically instead. Fix: describeSituation() now computes whether cache is elevated (CACHE_ELEVATED_THRESHOLD_MB = 100) in code and tells the model an unambiguous conclusion — "cache is elevated, call clear_cache" or "cache is nominal, do NOT call any tool" — rather than a number and a vague instruction to reason about it.

After all five fixes (structured-field fallback, multi-shape JSON parsing, markdown-fence stripping, pinned temperature, deterministic cache-threshold check), npm run eval:full scores 20/20 against qwen2.5-coder:7b. 14 of those 20 still go through the fallback text parser — this model reliably reasons correctly but has not been observed emitting Ollama's structured tool_calls field even once across ~80 calls made while building this harness, so the fallback path is load-bearing, not a rare edge case.

Tradeoffs / design decisions

• SQLite over a real time-series DB: sufficient at this scale (single host, 50-row sliding window), avoids an extra service dependency for a local tool.

• Polling (4s) over OS-level event hooks: simpler, portable across platforms; costs responsiveness for very short-lived failures between ticks.

• Cooldown-gated agent spawn (30s) instead of a queue: prevents cascading agent loops when multiple services degrade at once, at the cost of possibly missing a fix window if a new issue appears mid-cooldown.

• Local LLM (Ollama) over hosted API: zero marginal cost and no data leaving the host, at the cost of weaker reasoning than a frontier hosted model — mitigated by keeping the tool surface small and explicit rather than relying on open-ended reasoning.

• Separate child process per agent run rather than an in-process call: isolates a slow or hung model call from the main event loop and dashboard responsiveness.

Known limitations

• Remediation actions (restart_redis, restart_postgres, clear_cache) are a fixed, small action set — this is intentionally scoped as a proof of concept, not a general-purpose ops agent.

• No test suite yet; the mcp-server test script is a placeholder.

• n8n webhook forwarding degrades silently if n8n isn't running locally — this is expected in a standalone demo, but worth knowing before assuming remediation events are always externally visible.

Evals

mcp-server/src/evals/ scores the remediation agent's decisions against 20 synthetic telemetry scenarios (cases.ts) — single degraded services, multiple simultaneous degradations, healthy-but-high-resource states, and a no-data edge case.

Two stages, run via npm run eval (or npm test) inside mcp-server/:

1. Prompt-construction check (always runs, no LLM call, no Ollama required) — verifies describeSituation() actually surfaces the facts the model needs (which service is degraded, current CPU/RAM/cache) for every case. Deterministic, safe for CI.

2. Full LLM-graded run (npm run eval:full) — calls the real model via agentCore.decide() with execute: false (scores the decision, fires no docker restarts or webhooks) and checks whether the returned tool call matches the expected action. Requires a local Ollama instance running qwen2.5-coder:7b.

Note on ground truth: the degraded-service cases have an unambiguous correct action. The cache-threshold cases (e.g. "150MB, no degraded service, should call clear_cache") encode a judgment call rather than a hard rule, since the system prompt intentionally leaves the threshold fuzzy ("elevated," "far above normal") rather than hardcoding a number — worth knowing before treating a miss on those specific cases as a regression.

Running it

Requires Ollama running locally with qwen2.5-coder:7b pulled, and Docker running if you want the docker-status check and postgres remediation to do anything real.

# terminal 1 — MCP/API server
cd mcp-server
npm install
npm start

# terminal 2 — dashboard
cd react-dashboard
npm install
npm run dev

Dashboard: http://localhost:5173. API/SSE: http://localhost:3001.