mcp-incident-copilot

An MCP server that gives AI agents guarded, audited, read-only access to ops tooling (alerts, metrics, logs, deploys, runbooks), plus a triage agent that diagnoses an injected Kafka-lag incident end to end in 7 tool calls, verified against ground truth in CI, with the full transcript committed.

What this solves

• Agents helping with incidents need access to logs and metrics, and handing them raw credentials is how "AI-assisted ops" fails security review; here every capability is a schema-validated, read-only MCP tool with an audit trail, and the agent has no side door by construction (ADR-001).

• Post-incident "how did we conclude that" is usually archaeology; here the transcript of every tool call, with the agent's stated reasoning per step, is the committed artifact of the diagnosis.

• Correlation-based triage is easy to get confidently wrong; this repo's war story is its own agent blaming a deploy 35 minutes late because its inflection detector found where the metric grew fastest instead of where the regime changed, caught by a ground-truth test and fixed with CUSUM.

Related MCP server: mcp-incident-responder

Why this exists

Two things are true at once: agents are genuinely useful in incident triage (they read faster, correlate wider, and never forget to check deploy markers), and no responsible platform team will hand an agent Grafana admin and kubectl. The resolution is a governed tool surface. This repo builds one over MCP: five read-only tools (list_alerts, list_deploys, query_metric, search_logs, get_runbook), each with a declared schema, argument validation that rejects unknown and missing arguments as typed JSON-RPC errors, result shapes designed for a context window, and an audit trail that becomes part of the incident record.

On top of it runs a triage agent that speaks only MCP, one call at a time. The local planner is a deterministic policy encoding the generic-triage runbook (alerts first, then the timeline of deploys and metric inflections, then mechanism evidence from logs, then the prescribed action); LangGraph over Azure OpenAI is the production planner behind the same interface (ADR-002). Determinism is what buys the strongest claim in the repo: the incident environment is seeded with ground truth (a checkout-consumer deploy at minute 60 introduces a slow deserializer; lag ramps; payment errors follow; pages fire at 74 and 81), and CI fails the build if the agent's diagnosis names the wrong service. The whole loop, transcript, and diagnosis are reproducible from one command.

The incident and the diagnosis

The recorded triage, step by step (this is the actual committed out/transcript.json rendered):

Tech stack

Quickstart

Prerequisites: Python 3.11+, git.

git clone https://github.com/<you>/mcp-incident-copilot.git
cd mcp-incident-copilot
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

pytest --cov=copilot          # includes the ground-truth diagnosis test

python -m copilot.cli         # run the triage; writes out/transcript.json + out/diagnosis.json
cat out/diagnosis.json

Point any MCP client at McpOpsServer to use the same tools interactively; the server speaks standard initialize / tools/list / tools/call.

The numbers

Architecture decisions

ADR-001: the in-repo agent goes through MCP too; a governance layer with a side door is decoration. ADR-002: the boring choice of a deterministic runbook policy locally with LangGraph/Azure OpenAI as the prod planner, and why that is what makes the diagnosis CI-assertable.

Intentionally out of scope

• Write actions (restarts, rollbacks, scaling). Read-only is a security posture, not a limitation; a write tool needs approvals and idempotency design of its own. Trigger: a human-in-the-loop approval flow.

• Live Prometheus/Loki adapters. The tool interfaces are shaped for them (range summaries, capped log results); adapters are mechanical and listed first in future work.

• LLM-planner demo recordings. A canned LLM transcript would demo nothing verifiable; the deterministic transcript is complete and honest (ADR-002).

Security and compliance

The agent holds no credentials to any underlying system: it can only call declared MCP tools, all read-only. Argument validation rejects anything outside the schema. Every call is audited in order, and the audit is part of the incident artifact. Log results are capped and pre-structured; nothing streams raw production data into a model context unbounded.

Failure modes

Hardest problem solved

The agent's first full run produced a confident, well-evidenced, wrong answer: it blamed the notifications deploy at minute 95. Ground truth was the checkout-consumer deploy at minute 60. Every step in the transcript looked reasonable; the failure was one number: query_metric reported the lag inflection at minute 92.

The detector computed inflection as the argmax of the smoothed slope. For a metric that goes flat-then-ramp, the slope after onset is roughly constant with noise, so the largest smoothed slope lands anywhere inside the ramp, and this replay put it 32 minutes late, right next to an innocent deploy. The correlation logic then did its job correctly on a wrong input, which is what made the output convincing.

The fix (commit fix(tools): detect metric regime onset with CUSUM, not max slope) reframes the question: not "where does the series move fastest" but "where does the slope regime change". CUSUM over the centered first differences answers that: the cumulative sum falls while the slope is below its mean and rises after, so its extremum is the onset. Measured: lag onset 59, error-rate onset 60, against an injected fault at 60, stable within ±3 minutes across four seeds (tested). A strength score gates flat series as insignificant so the planner cannot build a timeline out of noise. The general lesson: in incident tooling, the analytical primitive underneath the reasoning, not the reasoning itself, is where confident wrongness comes from, and ground-truth tests are how you catch it.

Future work

• Prometheus and Loki adapters behind the existing tool interfaces.

• A human-approval write tool (propose_rollback) that files the action for confirmation rather than executing.

• LangGraph planner wiring as the prod profile with per-step token budgets.

• Multi-incident environments (concurrent faults) to stress the timeline logic.

• First metric to watch in adoption: transcript length distribution. Rising call counts per diagnosis means the tool results are not carrying enough signal per call.

License

MIT