MCP Incident Commander

A safe, stateful multi-agent workflow for investigating service incidents. The system gathers evidence through read-only MCP tools, correlates logs, metrics, and deployments, proposes a remediation, pauses for human approval, and creates a postmortem.

The default demo is deterministic and runs without cloud credentials or an LLM key. Optional LangGraph and MCP integrations show how the same workflow can be connected to production systems.

Why this project

Incident response is a strong agentic use case because useful answers require multiple tools, evidence correlation, state, and strict safety boundaries. This project demonstrates that an LLM may recommend an action, but a deterministic workflow controls which tools run and requires approval before a consequential step.

Related MCP server: Nibble

Features

• Specialized triage, evidence, root-cause, remediation, and postmortem agents

• Read-only MCP tools for metrics, logs, and deployment history

• Evidence-backed hypotheses with confidence and cited signals

• Human-in-the-loop approval gate; no automatic remediation

• Durable JSON checkpoints after each workflow stage

• Dry-run action executor and complete audit history

• Optional LangGraph orchestration and FastAPI control plane

• Docker packaging, tests, and GitHub Actions CI

Architecture

flowchart LR
    A[Alert webhook] --> B[Triage agent]
    B --> C[Evidence agent]
    C --> T1[MCP: metrics]
    C --> T2[MCP: logs]
    C --> T3[MCP: deployments]
    T1 & T2 & T3 --> D[Root-cause agent]
    D --> E[Remediation planner]
    E --> F{Human approval}
    F -->|reject| G[Close safely]
    F -->|approve| H[Dry-run executor]
    H --> I[Postmortem agent]
    B & C & D & E & F & H & I --> J[(Checkpoint + audit log)]

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Investigate and stop at the approval gate
incident-commander --incident examples/db-latency.json

# Demonstrate the approved dry-run path
incident-commander \
  --incident examples/db-latency.json \
  --approve \
  --approved-by demo-user

Run tests:

pytest -q

Example finding

For the included database-latency incident, the evidence layer returns:

• a sharp p95 latency increase;

• repeated database connection timeouts;

• a recent deployment that reduced the connection-pool limit.

The root-cause agent correlates those independent signals, assigns confidence, and proposes rolling back the relevant configuration. The workflow still stops until a human explicitly approves it.

MCP server

Install the optional agentic dependencies:

pip install -e ".[agentic]"
mcp run src/incident_commander/mcp_server.py --transport streamable-http

The server exposes only read operations:

• get_service_metrics

• search_service_logs

• get_recent_deployments

No shell or mutation tool is exposed to the model.

API control plane

pip install -e ".[api]"
uvicorn incident_commander.api:app --reload

Routes:

• POST /incidents/investigate

• POST /incidents/{incident_id}/approve

• GET /health

LangGraph integration

langgraph_workflow.py maps the deterministic agents to an explicit state graph. The core workflow remains framework-independent so safety policies and tests do not depend on an LLM framework.

Safety model

1. Evidence tools are read-only.

2. Every finding contains the signals that support it.

3. Proposed changes are data, not automatically executed commands.

4. Approval identity and stage transitions are recorded.

5. The included executor is always a dry run.

Production extensions

• Replace demo tools with Prometheus, Loki, Kubernetes, and GitHub MCP servers

• Store checkpoints in PostgreSQL or Redis

• Emit OpenTelemetry spans for every agent and tool call

• Add policy-as-code authorization for tools and services

• Open a remediation pull request instead of executing infrastructure commands

• Evaluate RCA accuracy against a labeled incident dataset

License

MIT