# MCP Incident Commander

A safe, stateful multi-agent workflow for investigating service incidents. The
system gathers evidence through read-only MCP tools, correlates logs, metrics,
and deployments, proposes a remediation, pauses for human approval, and creates
a postmortem.

The default demo is deterministic and runs without cloud credentials or an LLM
key. Optional LangGraph and MCP integrations show how the same workflow can be
connected to production systems.

## Why this project

Incident response is a strong agentic use case because useful answers require
multiple tools, evidence correlation, state, and strict safety boundaries. This
project demonstrates that an LLM may recommend an action, but a deterministic
workflow controls which tools run and requires approval before a consequential
step.

## Features

- Specialized triage, evidence, root-cause, remediation, and postmortem agents
- Read-only MCP tools for metrics, logs, and deployment history
- Evidence-backed hypotheses with confidence and cited signals
- Human-in-the-loop approval gate; no automatic remediation
- Durable JSON checkpoints after each workflow stage
- Dry-run action executor and complete audit history
- Optional LangGraph orchestration and FastAPI control plane
- Docker packaging, tests, and GitHub Actions CI

## Architecture

```mermaid
flowchart LR
    A[Alert webhook] --> B[Triage agent]
    B --> C[Evidence agent]
    C --> T1[MCP: metrics]
    C --> T2[MCP: logs]
    C --> T3[MCP: deployments]
    T1 & T2 & T3 --> D[Root-cause agent]
    D --> E[Remediation planner]
    E --> F{Human approval}
    F -->|reject| G[Close safely]
    F -->|approve| H[Dry-run executor]
    H --> I[Postmortem agent]
    B & C & D & E & F & H & I --> J[(Checkpoint + audit log)]
```

## Quick start

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Investigate and stop at the approval gate
incident-commander --incident examples/db-latency.json

# Demonstrate the approved dry-run path
incident-commander \
  --incident examples/db-latency.json \
  --approve \
  --approved-by demo-user
```

Run tests:

```bash
pytest -q
```

## Example finding

For the included database-latency incident, the evidence layer returns:

- a sharp p95 latency increase;
- repeated database connection timeouts;
- a recent deployment that reduced the connection-pool limit.

The root-cause agent correlates those independent signals, assigns confidence,
and proposes rolling back the relevant configuration. The workflow still stops
until a human explicitly approves it.

## MCP server

Install the optional agentic dependencies:

```bash
pip install -e ".[agentic]"
mcp run src/incident_commander/mcp_server.py --transport streamable-http
```

The server exposes only read operations:

- `get_service_metrics`
- `search_service_logs`
- `get_recent_deployments`

No shell or mutation tool is exposed to the model.

## API control plane

```bash
pip install -e ".[api]"
uvicorn incident_commander.api:app --reload
```

Routes:

- `POST /incidents/investigate`
- `POST /incidents/{incident_id}/approve`
- `GET /health`

## LangGraph integration

`langgraph_workflow.py` maps the deterministic agents to an explicit state
graph. The core workflow remains framework-independent so safety policies and
tests do not depend on an LLM framework.

## Safety model

1. Evidence tools are read-only.
2. Every finding contains the signals that support it.
3. Proposed changes are data, not automatically executed commands.
4. Approval identity and stage transitions are recorded.
5. The included executor is always a dry run.

## Production extensions

- Replace demo tools with Prometheus, Loki, Kubernetes, and GitHub MCP servers
- Store checkpoints in PostgreSQL or Redis
- Emit OpenTelemetry spans for every agent and tool call
- Add policy-as-code authorization for tools and services
- Open a remediation pull request instead of executing infrastructure commands
- Evaluate RCA accuracy against a labeled incident dataset

## License

MIT