incident-triage
README.md
# Incident Triage MCP Server
An AI-assisted **Incident Triage MCP Server** that helps engineers investigate service incidents using semantic log search, error aggregation, RAG-based diagnosis, and runbook recommendations.
The project demonstrates how **Model Context Protocol (MCP)** can be applied to a real-world operational engineering workflow rather than a generic chatbot.
The server operates against synthetic messaging/CPaaS service logs and exposes four MCP tools that can be invoked from an MCP client or directly from the terminal using the FastMCP CLI.
---
## 🚀 What This Project Does
Instead of manually searching through hundreds of log lines, an engineer can ask questions such as:
```text
Why is messaging-service failing?
Find logs related to queue backlog.
What's the runbook for a connection timeout?
What is the likely root cause of the messaging-service incident?
```
The MCP server converts these requests into structured tool calls against the incident data.
The important distinction is that the system **does not ask an LLM to guess from nothing**.
It combines:
* Deterministic log processing
* Semantic search
* Structured error aggregation
* Retrieval-Augmented Generation (RAG)
* LLM-based incident analysis
* Semantic runbook matching
---
# 🏗️ Architecture
```text
┌─────────────────────────┐
│ MCP Client │
│ │
│ Claude / FastMCP CLI │
└────────────┬────────────┘
│
│ MCP / stdio
▼
┌─────────────────────────┐
│ Incident Triage │
│ MCP Server │
│ server.py │
└────────────┬────────────┘
│
┌───────────────────────┼───────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Log Search │ │ Error │ │ Runbook │
│ │ │ Summary │ │ Search │
│ Embeddings │ │ Plain Code │ │ Embeddings │
└──────┬───────┘ └──────────────┘ └──────┬───────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ logs.jsonl │ │runbooks.json │
└──────────────┘ └──────────────┘
│
▼
┌─────────────────┐
│ Incident │
│ Diagnosis │
│ │
│ Retrieve Logs │
│ ↓ │
│ Anthropic LLM │
│ ↓ │
│ Root Cause + │
│ Next Steps │
└─────────────────┘
```
---
# 🔧 MCP Tools
The server exposes four tools.
## 1. `search_logs`
Performs semantic search over service logs using `sentence-transformers`.
Example:
```bash
fastmcp call server.py search_logs query="queue backlog"
```
The query is converted into an embedding and compared with log embeddings.
This allows queries such as:
```text
queue is stuck
```
to find logs containing concepts such as:
```text
consumer lag
queue backlog
```
without depending entirely on exact keyword matching.
---
## 2. `get_error_summary`
Provides a deterministic aggregation of errors.
Example:
```bash
fastmcp call server.py get_error_summary
```
Example result:
```text
Error summary:
db_deadlock: 38
rate_limit_exceeded: 37
queue_backlog: 37
auth_token_expired: 32
connection_timeout: 24
```
No LLM is involved here.
This is intentional.
Simple counting does not require AI.
---
## 3. `diagnose_incident`
Performs RAG-based incident diagnosis.
Example:
```bash
fastmcp call server.py diagnose_incident \
service="messaging-service" \
hours=24
```
The flow is:
```text
Service
│
▼
Retrieve recent errors
│
▼
Build incident context
│
▼
Send context to Anthropic
│
▼
Root cause hypothesis
│
▼
Immediate next steps
```
The LLM receives retrieved incident evidence rather than being asked to diagnose the problem without context.
If `ANTHROPIC_API_KEY` is not available, the POC falls back to a raw error-frequency summary.
---
## 4. `suggest_runbook`
Uses semantic similarity to find the most relevant operational runbooks.
Example:
```bash
fastmcp call server.py suggest_runbook \
query="connection to provider keeps timing out"
```
Example:
```text
Downstream Connection Timeout
Provider Auth Token Expired
Provider Rate Limiting
```
This allows engineers to describe an incident naturally instead of remembering the exact runbook title.
---
# 🧠 AI vs Traditional Logic
One of the goals of this POC is to demonstrate that **not everything needs an LLM**.
| Capability | Approach | Why |
| ------------------- | ------------------ | ----------------------------------- |
| `search_logs` | Embeddings | Semantic matching |
| `get_error_summary` | Python aggregation | No AI required |
| `diagnose_incident` | RAG + Anthropic | Converts evidence into a diagnosis |
| `suggest_runbook` | Embeddings | Handles different incident phrasing |
This creates a practical hybrid architecture:
```text
Incident Triage
│
┌────────────┼────────────┐
│ │ │
Deterministic Semantic LLM
Logic Retrieval Reasoning
│ │ │
Aggregation Embeddings RAG
```
---
# 📁 Project Structure
```text
incident-mcp/
│
├── server.py
├── requirements.txt
├── README.md
├── .gitignore
│
└── data/
├── generate_logs.py
└── runbooks.json
```
`logs.jsonl` is generated locally and should not be committed to Git.
---
# 🛠️ Technology Stack
* Python 3.12
* FastMCP
* Model Context Protocol
* Sentence Transformers
* `all-MiniLM-L6-v2`
* PyTorch
* NumPy
* SciPy
* Anthropic API
* python-dotenv
* JSONL
* Synthetic operational logs
---
# ⚙️ Local Setup
## 1. Clone the repository
```bash
git clone git@github.com:balachoudry-tech/incident_mcp_ai.git
cd incident_mcp_ai
```
---
## 2. Create a virtual environment
Python 3.12 is recommended.
```bash
python3.12 -m venv .venv
```
Activate it:
### macOS / Linux
```bash
source .venv/bin/activate
```
### Windows
```powershell
.venv\Scripts\activate
```
---
## 3. Install dependencies
```bash
pip install -r requirements.txt
```
The dependency versions are pinned to the versions used by the working POC environment.
---
# 📊 Generate Synthetic Logs
Generate the incident data:
```bash
python data/generate_logs.py
```
This creates:
```text
data/logs.jsonl
```
The generated logs simulate operational failures across messaging/CPaaS services.
---
# 🔐 Configure Anthropic
Create a `.env` file in the project root:
```env
ANTHROPIC_API_KEY=your_api_key_here
```
The `.env` file is intentionally excluded from Git.
The application loads the key using `python-dotenv`.
Never commit your API key.
---
# ▶️ Run the MCP Server
The server uses **stdio transport**.
```bash
python server.py
```
When started directly, the server waits for an MCP client.
You may see:
```text
Starting MCP server 'incident-triage'
with transport 'stdio'
```
This is expected.
The server is not a traditional HTTP API.
---
# 🧪 Test from Terminal
The POC can be tested without Claude Desktop using the FastMCP CLI.
## List available tools
```bash
fastmcp list server.py
```
Expected tools:
```text
search_logs
get_error_summary
diagnose_incident
suggest_runbook
```
---
## Test error summary
```bash
fastmcp call server.py get_error_summary
```
---
## Test semantic search
```bash
fastmcp call server.py search_logs \
query="queue backlog"
```
You can also specify the number of results:
```bash
fastmcp call server.py search_logs \
query="queue backlog" \
top_k=5
```
---
## Test runbook search
```bash
fastmcp call server.py suggest_runbook \
query="connection to provider keeps timing out"
```
---
## Test incident diagnosis
```bash
fastmcp call server.py diagnose_incident \
service="messaging-service" \
hours=24
```
With `ANTHROPIC_API_KEY` configured, this executes the RAG + Anthropic diagnosis flow.
---
# 🔄 Incident Diagnosis Flow
A typical diagnosis follows this flow:
```text
1. Engineer asks about a service
│
▼
2. Retrieve recent service errors
│
▼
3. Build incident context
│
▼
4. Send relevant context to LLM
│
▼
5. LLM analyzes relationships
│
▼
6. Generate root-cause hypothesis
│
▼
7. Recommend immediate next steps
```
For example:
```text
Carrier timeout
│
▼
Delivery retries
│
▼
Rate limiting
│
▼
Retry contention
│
▼
Database deadlocks
│
▼
Queue backlog
```
The LLM can use these correlated signals to produce a higher-level incident hypothesis.
---
# 📦 Why No Vector Database?
This POC intentionally does not use a vector database.
The log dataset is small enough that embeddings can be:
```text
Loaded
↓
Computed
↓
Stored in memory
↓
Compared using similarity
```
For a small proof of concept, this keeps the architecture simple.
A persistent vector index would become useful as the dataset grows.
---
# 🔒 Security Notes
This is a proof of concept.
The following production concerns are intentionally outside the scope:
* Authentication
* Authorization
* Multi-tenancy
* Secret management
* Tool-level permissions
* Audit logging
* Production observability
* Persistent vector storage
* Real log ingestion
* Production-grade error handling
API keys should never be committed to the repository.
---
# 🚀 Production Evolution
The POC can later evolve toward:
```text
┌──────────────────────┐
│ Real Log Sources │
│ │
│ CloudWatch │
│ Elasticsearch │
│ Loki │
│ OpenSearch │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Log Processing │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Persistent Vector DB │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ MCP Server │
│ │
│ Search │
│ Summarize │
│ Diagnose │
│ Runbooks │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ MCP Client │
│ / Incident Platform │
└──────────────────────┘
```
Potential future improvements include:
* Real production log sources
* Persistent vector indexes
* Authentication and authorization
* Tool-level access control
* Structured tracing
* Incident correlation IDs
* Confidence scoring
* Better "insufficient evidence" handling
* Audit trails
* Production observability
* Multi-service incident correlation
---
# 🎯 What This POC Demonstrates
This project demonstrates several practical AI engineering patterns:
### MCP
Building operational capabilities as reusable MCP tools.
### Semantic Search
Using embeddings to search logs based on meaning rather than exact keywords.
### RAG
Retrieving relevant operational evidence before asking an LLM to reason about an incident.
### Tool Selection
Using deterministic code where deterministic code is sufficient and AI where reasoning provides additional value.
### Operational AI
Applying GenAI to an engineering workflow where the output is grounded in actual incident evidence.
---
# ⚠️ Disclaimer
The logs and incidents in this repository are **synthetic** and are intended only for demonstration and learning purposes.
The diagnosis generated by the LLM should not be treated as an authoritative production incident response.
---
# 📄 License
Add the project's license here if/when one is selected.
---
## Author
Built as an engineering-focused MCP/RAG POC demonstrating AI-assisted incident triage for high-throughput messaging/CPaaS systems.
This server cannot be deployed
Maintenance
ActivitySlowing
ResponsivenessNo issues