# 🩺 IncidentIQ

### An AI on-call engineer that actually checks its work before it blames anyone.

<p>
  <img alt="Python" src="https://img.shields.io/badge/python-3.10%2B-3776AB?logo=python&logoColor=white">
  <img alt="Claude Opus 5" src="https://img.shields.io/badge/LLM-Claude%20Opus%205-8B5CF6?logo=anthropic&logoColor=white">
  <img alt="MCP" src="https://img.shields.io/badge/protocol-MCP-22D3EE">
  <img alt="RAG" src="https://img.shields.io/badge/RAG-Chroma%20%2B%20MiniLM--L6-34D399">
  <img alt="Streamlit" src="https://img.shields.io/badge/UI-Streamlit-FF4B4B?logo=streamlit&logoColor=white">
  <img alt="Status" src="https://img.shields.io/badge/status-working%20demo-success">
</p>

---

## The 30-second pitch

Imagine payments on your site just started failing. You'd want an on-call
engineer who does two things at once: **checks what's actually happening
right now** (is the database okay? did someone deploy something?), and
**remembers what happened last time** something like this occurred — without
blindly assuming this time is the same.

**IncidentIQ is that engineer, built as an AI agent.** Give it a one-line
description of an incident, and it investigates on its own — deciding
which tool to reach for next based on what it just learned, the same way
a real SRE would — until it has enough evidence to hand you a root cause,
not a guess.

It's a from-scratch build of the two ideas everyone's talking about right
now, **RAG** and **MCP**, done deliberately slowly over about a week so
every layer could be understood and tested on its own before the next one
got stacked on top. Not a framework demo. Not a tutorial copy-paste. Six
days, six testable layers, one real bug found and fixed by actually
running the thing.

<details>
<summary><b>New to "RAG" and "MCP"? Click here for the 10-second version</b></summary>
<br>

- **MCP (Model Context Protocol)** is just a standard way to hand an AI
  agent a set of *tools* — think "check the database," "read the logs" —
  the same way a human on-call engineer would open a few dashboards.
- **RAG (Retrieval-Augmented Generation)** is how the agent gets a
  *memory* — a searchable library of past incident reports it can query
  for precedent, instead of only knowing what's in its training data.

The interesting part isn't building either one — it's realizing the agent
shouldn't have to think about them differently. See
[**"How it thinks"**](#-how-it-thinks-the-one-design-decision-that-matters) below.

</details>

## 👀 See it in action

Real, unedited output — not a mockup — from asking it:

> *"Payments started failing around 2:30 PM. What could be the cause?"*

```
🔧 get_service_health(payment-service) → down, error_rate 0.87
🔧 get_service_health(auth/order/database/kafka)  → 4 healthy-or-degraded reads
🔧 get_database_status()          → connection pool 99% exhausted since 14:25
🔧 get_recent_logs(payment-service, since=14:15) → deploy log line found
🔧 get_deployment_info(payment-service) → dep-1045: pool size cut 50 → 10
🔧 get_deployment_info(order-service, database)  → ruling out other causes
📚 search_incident_history("connection pool exhaustion...")
   → finds a *similar-looking* past incident with a DIFFERENT cause

💬 Root cause: dep-1045 (14:15:00) cut payment-service's DB connection
   pool from 50 → 10. Undersized pool couldn't serve normal traffic,
   requests queued and timed out, retry storm saturated the database.

   Ruled out: traffic spike (the cause of a similar-looking past
   incident) — request volume was flat before AND after the deploy,
   so this isn't that.

   Confidence: High.  →  4 iterations, 12 tool calls, 0 errors.
```

The agent retrieved a past incident that looked almost identical — then
checked live evidence, decided that incident's explanation *didn't* apply
here, and said so explicitly. That's not an accident; it's the one design
decision this whole project is built around. Full trace, full
walkthrough, and where the agent could've dug one level deeper:
[**Example run**](#-example-run-real-trace).

## ⚡ Quick start

```bash
git clone <this-repo> && cd IncidentIQ
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python build_index.py                 # embeds the knowledge base (~90MB model, one-time)
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env

streamlit run app.py                  # the UI, or:
python run_agent.py                   # ...the terminal version
```

That's it — no Docker, no external services, no real infrastructure.
Everything the agent investigates (`world/`) and everything it can
research (`docs/`) is fake, hand-built, and lives in this repo. See
[**Running it**](#️-running-it) for the full breakdown, including how to
test each layer on its own.

## 🧠 How it thinks: the one design decision that matters

To the agent, **there is no difference between "check what's happening
right now" and "search what happened before."** Both are just entries in
one list of 6 tools — same server, same calling convention, same
description style. That's enforced at the code level, not just a nice
idea: `search_incident_history` (RAG) sits in the exact same MCP server
as the five live-state tools, registered the same way. If retrieval and
live tools lived in separate systems, the agent would have to reason
about *which subsystem* to talk to — a distinction that matters to the
people building this, but has no business leaking into the agent's own
judgment.

**But the two kinds of information don't earn the same trust.** Live
tools report the current, actual truth of the system. A retrieved past
incident is only ever a *lead* — because the knowledge base contains a
deliberate trap: `INC-2024-0417` looks, at the symptom level, identical
to the live incident (same "connection pool exhausted" signature) — but
its real cause was a traffic spike with *no deploy at all*, the opposite
of what's actually happening live. So the system prompt says, explicitly:
*retrieved history is a hypothesis to check, never a conclusion to
repeat — a root cause is only stated once live data confirms it.* The UI
reinforces the same idea visually, tagging every step 🔧 (live) or 📚
(historical) so the distinction is legible to a human watching, not just
buried in a prompt nobody reads.

This got tested for real, not just designed on paper — the trap fired in
the actual run above, and the agent didn't fall for it. See the
walkthrough in [**Example run**](#-example-run-real-trace).

## 🏗️ Architecture

```mermaid
flowchart TB
    subgraph World["world/ — fake production state (JSON + logs)"]
        services["services.json"]
        logs["logs/*.log"]
        deploys["deployments.json"]
        dbstatus["database_status.json"]
    end

    subgraph Docs["docs/ — knowledge base (markdown)"]
        incidents["3 past incident postmortems"]
        runbook["1 runbook"]
        arch["1 architecture doc"]
        api["1 API reference"]
    end

    subgraph RAGPipeline["rag/ — retrieval pipeline"]
        chunker["chunker.py — header-aware chunking"]
        store["store.py — embed (all-MiniLM-L6-v2) + store (Chroma)"]
    end
    Docs --> chunker --> store

    subgraph MCPServer["server/ — one MCP server, 6 tools"]
        data["data.py"]
        liveTools["🔧 get_service_health<br/>🔧 get_recent_logs<br/>🔧 get_kafka_consumer_lag<br/>🔧 get_deployment_info<br/>🔧 get_database_status"]
        ragTool["📚 search_incident_history"]
    end
    World --> data --> liveTools
    store --> ragTool

    subgraph AgentPkg["agent/ — orchestration"]
        bridge["mcp_client.py — MCP tools to Claude tool format"]
        loop["loop.py — manual agentic loop (Claude Opus 5)"]
    end
    liveTools --> bridge
    ragTool --> bridge
    bridge --> loop

    subgraph Presentation["presentation only — no investigation logic"]
        cli["run_agent.py (CLI trace)"]
        streamlit["app.py (Streamlit UI)"]
    end
    loop --> cli
    loop --> streamlit
```

Nothing here is exotic — a JSON+log fixture, a vector DB, an MCP server,
a `while` loop calling the Claude API. The interesting part is the order
it got built in: every layer had to prove itself before the next one was
allowed to depend on it.

| Day | Built | Proven by |
|---|---|---|
| 1 | `world/` — fake services, logs, deploys, DB status | Manual inspection, consistent timestamps across every file |
| 2 | `server/` — MCP server, 5 live-state tools | `test_tools.py` — real MCP protocol, not bare function calls |
| 3 | `docs/` — knowledge base + chunking strategy | `chunk_docs.py` — chunk boundaries, table/code-fence integrity |
| 4 | `rag/` — embeddings + Chroma + `search_incident_history` | `test_retrieval.py` — retrieval quality graded query by query |
| 5 | `agent/` — the orchestration loop | Code-verified, then a real live run against the Claude API |
| 6 | `app.py` — Streamlit UI | Verified live in-browser, including a bug found and fixed mid-test |

## 📼 Example run (real trace)

Real captured output from `python run_agent.py` against the default
query, model `claude-opus-5`, unedited except for trimming a few long
tool results (marked `[truncated, N chars total]` — that's the CLI's own
behavior, not something removed for this README):

```
USER QUERY: Payments started failing around 2:30 PM. What could be the cause?
Connected to MCP server - 6 tools available:
  - get_service_health
  - get_recent_logs
  - get_kafka_consumer_lag
  - get_deployment_info
  - get_database_status
  - search_incident_history

==========================================================================================
ITERATION 1
==========================================================================================

💬 CLAUDE:
I'll start by checking health across all services to see the full picture.

🔧 TOOL CALL: get_service_health({"service": "payment-service"})
🔧 TOOL RESULT [OK]:
{"name": "payment-service", "status": "down", "error_rate": 0.87, "avg_latency_ms": 9800, ...}

🔧 TOOL CALL: get_service_health({"service": "auth-service"})
🔧 TOOL RESULT [OK]: {"name": "auth-service", "status": "healthy", "error_rate": 0.002, ...}

🔧 TOOL CALL: get_service_health({"service": "order-service"})
🔧 TOOL RESULT [OK]: {"name": "order-service", "status": "degraded", "error_rate": 0.34, ...}

🔧 TOOL CALL: get_service_health({"service": "database"})
🔧 TOOL RESULT [OK]: {"name": "database", "status": "degraded", "error_rate": 0.51, ...}

🔧 TOOL CALL: get_service_health({"service": "kafka"})
🔧 TOOL RESULT [OK]: {"name": "kafka", "status": "healthy", "error_rate": 0.001, ...}

==========================================================================================
ITERATION 2
==========================================================================================

🧠 THINKING:
Payment is down, order and database are degraded while auth and kafka remain
healthy. I should check the DB pool status and logs, along with recent
deployments, to trace the cause.

💬 CLAUDE:
Payment-service is down, order-service and database are degraded, auth and
kafka are fine. Let me dig into the database pool and payment logs, plus
recent deploys.

🔧 TOOL CALL: get_database_status({})
🔧 TOOL RESULT [OK]:
{
  "database": "orders-primary-pg", "current_status": "degraded",
  "incident_started_at": "2026-09-05T14:25:00-05:00",
  "connection_pool": {"max_connections": 200, "active_connections": 198,
                       "waiting_requests": 143, "utilization_pct": 99},
  "history": [
    {"timestamp": "...14:15:00...", "note": "payment-service deployed v4.2.1 (pool max reduced 50->10 on client side)"},
    {"timestamp": "...14:23:00...", "note": "payment-service connection pool (max 10) saturated, requests queuing on client side"},
    ...
  ]
}

🔧 TOOL CALL: get_recent_logs({"service": "payment-service", "since": "14:15"})
🔧 TOOL RESULT [OK]: {"count": 155, "logs": [
  {"timestamp": "2026-09-05 14:15:03", "level": "INFO", "message": "Deployment complete: version=v4.2.1 (config: db_pool.max_size 50 -> 10)"},
  ... [truncated, 25820 chars total]

🔧 TOOL CALL: get_deployment_info({"service": "payment-service"})
🔧 TOOL RESULT [OK]:
{"service": "payment-service", "count": 1, "deployments": [
  {"id": "dep-1045", "timestamp": "2026-09-05T14:15:00-05:00", "version": "v4.2.1",
   "change_description": "Reduce DB connection pool max size from 50 to 10 to lower idle connection count per cost-optimization ticket PERF-881",
   "deployed_by": "arjun.kapoor", "status": "success"}
]}

==========================================================================================
ITERATION 3
==========================================================================================

🧠 THINKING:
This is compelling but I should rule out alternative explanations before
concluding. I'll check order-service and database deployment history, and
search for similar past incidents, to make sure I'm not missing a competing
cause like a traffic spike.

🔧 TOOL CALL: get_deployment_info({"service": "order-service"})
🔧 TOOL RESULT [OK]: {"count": 1, "deployments": [{"id": "dep-1042", "timestamp": "2026-09-03T16:40:00-05:00", ...}]}

🔧 TOOL CALL: get_deployment_info({"service": "database"})
🔧 TOOL RESULT [OK]: {"count": 1, "deployments": [{"id": "dep-1044", "timestamp": "2026-09-05T06:00:00-05:00", ...}]}

📚 TOOL CALL: search_incident_history({"query": "payment-service database connection pool exhaustion connection timeout"})
📚 TOOL RESULT [OK]:
{"count": 5, "results": [
  {"similarity": 0.694, "source": "architecture/payment-service-database-kafka-architecture.md",
   "section": "... > payment-service ↔ database > Connection pooling",
   "content": "Both present identically at the symptom level - `connection timeout to database` errors ... so distinguishing them requires checking deployment history *and* traffic volume, not just the pool metrics alone."},
  {"similarity": 0.652, "source": "incidents/INC-2024-0417-payment-service-db-pool-exhaustion.md",
   "section": "... > Lessons learned",
   "content": "We correctly avoided the trap of assuming \"connection timeout to database\" always means a code/config regression ..."},
  ... [truncated, 5536 chars total]

🔧 TOOL CALL: get_recent_logs({"service": "database", "since": "14:20"})
🔧 TOOL RESULT [OK]: {"count": 41, "logs": [
  {"timestamp": "2026-09-05 14:20:00", "level": "WARN", "message": "orders-primary-pg connection pool utilization rising, active_connections=89"},
  ...
  {"timestamp": "2026-09-05 14:25:00", "level": "ERROR", "message": "orders-primary-pg connection pool exhausted: 160/200 connections active, 77 requests waiting"},
  ... [truncated, 7858 chars total]

==========================================================================================
ITERATION 4
==========================================================================================

🧠 THINKING:
Checking request cadence in the logs, traffic volume stayed steady before
and after 14:15, ruling out a traffic spike as the cause. I also confirm
the order-service retry deploy from Sep 3 amplified the issue. That's
enough evidence to move forward.

==========================================================================================
FINAL ANSWER (after 4 iteration(s), 12 tool call(s))
==========================================================================================
## Root cause

**A bad config change in payment-service deploy `dep-1045` (v4.2.1,
2026-09-05 14:15:00 CDT) that cut the client-side DB connection pool from
`max_size=50` to `max_size=10`.** The undersized pool couldn't serve normal
traffic, requests queued and timed out, and the retry storm then saturated
`orders-primary-pg`, dragging order-service down with it.

## Evidence (live data)

**The trigger — deploy at 14:15:**
- `dep-1045`, v4.2.1, deployed 14:15:00 by arjun.kapoor: "Reduce DB
  connection pool max size from 50 to 10 to lower idle connection count
  per cost-optimization ticket PERF-881" — status: success.
- payment-service log 14:15:03: `Deployment complete: version=v4.2.1
  (config: db_pool.max_size 50 -> 10)`.

**The causal chain, timestamp by timestamp:**
- 14:00–14:15 — healthy, 62–65 active connections, 0 waiting, 31–33% utilization.
- 14:20 — 71 active, 3 waiting: "Slight uptick in queued acquisitions from payment-service clients."
- 14:23 — 88 active, 19 waiting: pool "saturated, requests queuing on client side."
- 14:25 — 160 active, 77 waiting: clients "opening redundant retry connections."
- 14:40 (now) — 198/200 active, 143 waiting, 99% utilization, "no mitigation applied yet."

**Ruled out alternatives:**
- **Traffic spike** (the root cause of the very similar past incident
  INC-2024-0417, flash sale at 14:02): *not supported here.*
  Payment-service request cadence in the logs is flat from 14:15 onward —
  no volume increase. DB active connections were steady right up to the
  14:15 deploy. The rise starts at the deploy, not before it.
- **Other deploys:** order-service's only deploy is 2 days earlier;
  database's was a routine patch 8+ hours before onset. Neither correlates.
- **Auth/Kafka:** both healthy — not involved.

## Confidence

**High.** The deploy description, the payment-service log line, the error
message literally quoting `max_size=10`, and the connection-pool timeline
all independently converge on the same 14:15 config change.

## Suggested next steps

1. Roll back `dep-1045` / restore `db_pool.max_size` to 50.
2. Don't just restart payment-service — INC-2024-0417's lesson applies:
   restarts fix leaked connections, not undersized pools.
3. Consider briefly throttling order-service retries while the DB drains
   the 143-request backlog.
4. Follow up on PERF-881 with a pool size validated against actual
   concurrency, not a flat 5x reduction.
```

### Walkthrough — what the agent did, and where it fell short

**Iteration 1** was pure triage: all 5 `get_service_health` calls in
parallel, no assumptions. **Iteration 2** fired 3 more calls in parallel
(`get_database_status`, `get_recent_logs`, `get_deployment_info`) —
notably, the `since="14:15"` window was chosen as a reasonable generic
buffer *before* any of that batch's results came back (all three were
decided in one turn), and it happened to land exactly on the real deploy
time. **Iteration 3** made the single `search_incident_history` call of
the whole run, with a well-composed multi-concept query, then explicitly
worked to rule out order-service's and database's own deploys and check
whether traffic volume (not just a deploy) could explain it.

The moment I was watching for most closely: the RAG search's second
result was **INC-2024-0417 — the deliberately-planted trap incident**
with a *different* (traffic-spike) root cause. The final answer didn't
just retrieve it; it explicitly checked live evidence against it and
rejected it ("*Payments-service request cadence in the logs is flat...
not supported here*"). That's the RAG-grounding policy working exactly as
designed, several days after it was only a design decision on paper.

**Where it fell short:** order-service showed `degraded` (0.34 error
rate) in iteration 1, but the agent never called `get_recent_logs`
against order-service itself to verify it — it inferred order-service's
involvement entirely from a note embedded in the *database's* status
history ("order-service secondary read queries also beginning to
queue"). That note is accurate, so the conclusion wasn't wrong, but a
more thorough investigation would have pulled order-service's own logs
and seen the actual cascading 502/timeout pattern directly rather than
taking a secondhand mention as sufficient. The one clear "go one level
deeper" moment in an otherwise clean run.

**Totals:** 4 iterations, 12 tool calls, 0 errors, 0 duplicate calls, 0
fallbacks/refusals/truncation — well inside the 12-iteration budget.

## 🕹️ Running it

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Build the fake world's search index (only needed once, or after editing docs/)
python build_index.py

# CLI, full trace printed to the terminal
export ANTHROPIC_API_KEY=sk-ant-...
python run_agent.py "Payments started failing around 2:30 PM. What could be the cause?"

# Streamlit UI (also loads ANTHROPIC_API_KEY from a local .env if present,
# or you can paste a key directly into the page)
streamlit run app.py
```

Individual pieces can be exercised in isolation too — useful for trusting
each layer before trusting the whole:

```bash
python test_tools.py       # all 6 MCP tools, real protocol, no LLM involved
python test_retrieval.py   # RAG retrieval quality, graded query by query
```

## ⚠️ Known limitations

Being upfront about these rather than presenting this as production-ready:

- **It's entirely fake infrastructure.** `world/` and `docs/` are
  hand-authored fixtures, not real telemetry or a real incident corpus.
  Nothing here has been validated against a real production system.
- **The retrieval "trap" safety net is thinner than it looks.** As
  described above, the disambiguating "this was not a deploy" sentence in
  `INC-2024-0417` ranks outside top-5 for the queries most likely to need
  it (rank 8, similarity 0.410, for "database connection pool exhausted";
  absent from the top 10 entirely for "payment service errors after a
  deploy"). The system currently relies on (a) the architecture doc
  independently repeating similar disambiguating language, which happens
  to rank higher, and (b) the agent's system-prompt instruction to verify
  against live tools — not on retrieval ranking reliably surfacing the
  right counter-evidence on its own. If either the architecture doc's
  wording or the prompt instruction were removed, this could break
  silently.
- **The embedding model doesn't understand negation.** A chunk that
  narrates "checked deployment history... no deploys in the last 24h"
  still scores as a strong match for a query like "payment service errors
  after a deploy" — the embedding model matches on topical similarity, not
  on whether the passage confirms or rules out what's being asked.
- **`all-MiniLM-L6-v2` is a symmetric similarity model**, not tuned for
  asymmetric query→passage retrieval. Imperative, question-style queries
  ("how do I roll back a bad deploy") return much lower absolute
  similarity scores (~0.22-0.33) than descriptive queries (~0.5-0.7) even
  when the correct document still ranks first — so a similarity-threshold
  cutoff for "no good match found" would not be well-calibrated across
  different query phrasings.
- **A few chunks intentionally exceed the target chunk size.** 3 of 54
  chunks (incident report "Timeline" sections) are oversized because a
  bulleted list with no internal blank lines is one atomic block by
  design — kept whole on purpose, since splitting a timeline mid-sequence
  would lose the causal narrative an incident investigation actually
  needs, at the cost of a larger-than-target chunk.
- **`max_iterations=12`** is a hard cap on the agent loop with no smarter
  fallback — a genuinely hard investigation could be cut off before
  reaching a conclusion. The UI/CLI say so explicitly rather than
  silently truncating, but there's no partial-answer synthesis.
- **No memory across investigations.** Every query starts a fresh loop;
  nothing is remembered between runs or across a Streamlit page reload.
- **No systematic eval.** Retrieval and tool behavior were spot-checked
  against a handful of hand-picked queries (`test_retrieval.py`), not
  measured against a proper labeled eval set.
- **Streamlit app is single-user, local-only**, no auth, no persistence —
  a demo shell, not a deployable multi-user tool.

## 🎓 What building this actually taught me

- **Tool descriptions are the real interface.** An LLM never sees your
  code, your comments, or your intentions — only the `name`,
  `description`, and JSON schema you hand it. Half of this project's
  actual engineering was writing descriptions precise enough that a model
  reaching for the wrong tool became the exception, not the norm.
- **Retrieval quality has to be evaluated adversarially, not just happily.**
  It's easy to build a knowledge base where every test query finds a good
  match. It's much more informative to *plant a trap* — a document that's
  similar but wrong — and see whether the system (prompt + retrieval +
  agent judgment together) actually catches it. Ours did, but only barely:
  the safety net (see **Known limitations**) is thinner than the demo
  suggests.
- **A live run finds bugs a synthetic test can't.** The Streamlit final
  card looked perfect with hand-written sample data — headers, bold,
  bullets all styled correctly. Then a real model response, full of
  actual `##` and `**` markdown, exposed that `st.html()` doesn't parse
  markdown at all. Nothing about that would have shown up without
  actually running the real thing end to end.