🔧 get_service_health(payment-service) → down, error_rate 0.87
🔧 get_service_health(auth/order/database/kafka)  → 4 healthy-or-degraded reads
🔧 get_database_status()          → connection pool 99% exhausted since 14:25
🔧 get_recent_logs(payment-service, since=14:15) → deploy log line found
🔧 get_deployment_info(payment-service) → dep-1045: pool size cut 50 → 10
🔧 get_deployment_info(order-service, database)  → ruling out other causes
📚 search_incident_history("connection pool exhaustion...")
   → finds a *similar-looking* past incident with a DIFFERENT cause

💬 Root cause: dep-1045 (14:15:00) cut payment-service's DB connection
   pool from 50 → 10. Undersized pool couldn't serve normal traffic,
   requests queued and timed out, retry storm saturated the database.

   Ruled out: traffic spike (the cause of a similar-looking past
   incident) — request volume was flat before AND after the deploy,
   so this isn't that.

   Confidence: High.  →  4 iterations, 12 tool calls, 0 errors.

git clone <this-repo> && cd IncidentIQ
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python build_index.py                 # embeds the knowledge base (~90MB model, one-time)
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env

streamlit run app.py                  # the UI, or:
python run_agent.py                   # ...the terminal version

flowchart TB
    subgraph World["world/ — fake production state (JSON + logs)"]
        services["services.json"]
        logs["logs/*.log"]
        deploys["deployments.json"]
        dbstatus["database_status.json"]
    end

    subgraph Docs["docs/ — knowledge base (markdown)"]
        incidents["3 past incident postmortems"]
        runbook["1 runbook"]
        arch["1 architecture doc"]
        api["1 API reference"]
    end

    subgraph RAGPipeline["rag/ — retrieval pipeline"]
        chunker["chunker.py — header-aware chunking"]
        store["store.py — embed (all-MiniLM-L6-v2) + store (Chroma)"]
    end
    Docs --> chunker --> store

    subgraph MCPServer["server/ — one MCP server, 6 tools"]
        data["data.py"]
        liveTools["🔧 get_service_health<br/>🔧 get_recent_logs<br/>🔧 get_kafka_consumer_lag<br/>🔧 get_deployment_info<br/>🔧 get_database_status"]
        ragTool["📚 search_incident_history"]
    end
    World --> data --> liveTools
    store --> ragTool

    subgraph AgentPkg["agent/ — orchestration"]
        bridge["mcp_client.py — MCP tools to Claude tool format"]
        loop["loop.py — manual agentic loop (Claude Opus 5)"]
    end
    liveTools --> bridge
    ragTool --> bridge
    bridge --> loop

    subgraph Presentation["presentation only — no investigation logic"]
        cli["run_agent.py (CLI trace)"]
        streamlit["app.py (Streamlit UI)"]
    end
    loop --> cli
    loop --> streamlit

USER QUERY: Payments started failing around 2:30 PM. What could be the cause?
Connected to MCP server - 6 tools available:
  - get_service_health
  - get_recent_logs
  - get_kafka_consumer_lag
  - get_deployment_info
  - get_database_status
  - search_incident_history

==========================================================================================
ITERATION 1
==========================================================================================

💬 CLAUDE:
I'll start by checking health across all services to see the full picture.

🔧 TOOL CALL: get_service_health({"service": "payment-service"})
🔧 TOOL RESULT [OK]:
{"name": "payment-service", "status": "down", "error_rate": 0.87, "avg_latency_ms": 9800, ...}

🔧 TOOL CALL: get_service_health({"service": "auth-service"})
🔧 TOOL RESULT [OK]: {"name": "auth-service", "status": "healthy", "error_rate": 0.002, ...}

🔧 TOOL CALL: get_service_health({"service": "order-service"})
🔧 TOOL RESULT [OK]: {"name": "order-service", "status": "degraded", "error_rate": 0.34, ...}

🔧 TOOL CALL: get_service_health({"service": "database"})
🔧 TOOL RESULT [OK]: {"name": "database", "status": "degraded", "error_rate": 0.51, ...}

🔧 TOOL CALL: get_service_health({"service": "kafka"})
🔧 TOOL RESULT [OK]: {"name": "kafka", "status": "healthy", "error_rate": 0.001, ...}

==========================================================================================
ITERATION 2
==========================================================================================

🧠 THINKING:
Payment is down, order and database are degraded while auth and kafka remain
healthy. I should check the DB pool status and logs, along with recent
deployments, to trace the cause.

💬 CLAUDE:
Payment-service is down, order-service and database are degraded, auth and
kafka are fine. Let me dig into the database pool and payment logs, plus
recent deploys.

🔧 TOOL CALL: get_database_status({})
🔧 TOOL RESULT [OK]:
{
  "database": "orders-primary-pg", "current_status": "degraded",
  "incident_started_at": "2026-09-05T14:25:00-05:00",
  "connection_pool": {"max_connections": 200, "active_connections": 198,
                       "waiting_requests": 143, "utilization_pct": 99},
  "history": [
    {"timestamp": "...14:15:00...", "note": "payment-service deployed v4.2.1 (pool max reduced 50->10 on client side)"},
    {"timestamp": "...14:23:00...", "note": "payment-service connection pool (max 10) saturated, requests queuing on client side"},
    ...
  ]
}

🔧 TOOL CALL: get_recent_logs({"service": "payment-service", "since": "14:15"})
🔧 TOOL RESULT [OK]: {"count": 155, "logs": [
  {"timestamp": "2026-09-05 14:15:03", "level": "INFO", "message": "Deployment complete: version=v4.2.1 (config: db_pool.max_size 50 -> 10)"},
  ... [truncated, 25820 chars total]

🔧 TOOL CALL: get_deployment_info({"service": "payment-service"})
🔧 TOOL RESULT [OK]:
{"service": "payment-service", "count": 1, "deployments": [
  {"id": "dep-1045", "timestamp": "2026-09-05T14:15:00-05:00", "version": "v4.2.1",
   "change_description": "Reduce DB connection pool max size from 50 to 10 to lower idle connection count per cost-optimization ticket PERF-881",
   "deployed_by": "arjun.kapoor", "status": "success"}
]}

==========================================================================================
ITERATION 3
==========================================================================================

🧠 THINKING:
This is compelling but I should rule out alternative explanations before
concluding. I'll check order-service and database deployment history, and
search for similar past incidents, to make sure I'm not missing a competing
cause like a traffic spike.

🔧 TOOL CALL: get_deployment_info({"service": "order-service"})
🔧 TOOL RESULT [OK]: {"count": 1, "deployments": [{"id": "dep-1042", "timestamp": "2026-09-03T16:40:00-05:00", ...}]}

🔧 TOOL CALL: get_deployment_info({"service": "database"})
🔧 TOOL RESULT [OK]: {"count": 1, "deployments": [{"id": "dep-1044", "timestamp": "2026-09-05T06:00:00-05:00", ...}]}

📚 TOOL CALL: search_incident_history({"query": "payment-service database connection pool exhaustion connection timeout"})
📚 TOOL RESULT [OK]:
{"count": 5, "results": [
  {"similarity": 0.694, "source": "architecture/payment-service-database-kafka-architecture.md",
   "section": "... > payment-service ↔ database > Connection pooling",
   "content": "Both present identically at the symptom level - `connection timeout to database` errors ... so distinguishing them requires checking deployment history *and* traffic volume, not just the pool metrics alone."},
  {"similarity": 0.652, "source": "incidents/INC-2024-0417-payment-service-db-pool-exhaustion.md",
   "section": "... > Lessons learned",
   "content": "We correctly avoided the trap of assuming \"connection timeout to database\" always means a code/config regression ..."},
  ... [truncated, 5536 chars total]

🔧 TOOL CALL: get_recent_logs({"service": "database", "since": "14:20"})
🔧 TOOL RESULT [OK]: {"count": 41, "logs": [
  {"timestamp": "2026-09-05 14:20:00", "level": "WARN", "message": "orders-primary-pg connection pool utilization rising, active_connections=89"},
  ...
  {"timestamp": "2026-09-05 14:25:00", "level": "ERROR", "message": "orders-primary-pg connection pool exhausted: 160/200 connections active, 77 requests waiting"},
  ... [truncated, 7858 chars total]

==========================================================================================
ITERATION 4
==========================================================================================

🧠 THINKING:
Checking request cadence in the logs, traffic volume stayed steady before
and after 14:15, ruling out a traffic spike as the cause. I also confirm
the order-service retry deploy from Sep 3 amplified the issue. That's
enough evidence to move forward.

==========================================================================================
FINAL ANSWER (after 4 iteration(s), 12 tool call(s))
==========================================================================================
## Root cause

**A bad config change in payment-service deploy `dep-1045` (v4.2.1,
2026-09-05 14:15:00 CDT) that cut the client-side DB connection pool from
`max_size=50` to `max_size=10`.** The undersized pool couldn't serve normal
traffic, requests queued and timed out, and the retry storm then saturated
`orders-primary-pg`, dragging order-service down with it.

## Evidence (live data)

**The trigger — deploy at 14:15:**
- `dep-1045`, v4.2.1, deployed 14:15:00 by arjun.kapoor: "Reduce DB
  connection pool max size from 50 to 10 to lower idle connection count
  per cost-optimization ticket PERF-881" — status: success.
- payment-service log 14:15:03: `Deployment complete: version=v4.2.1
  (config: db_pool.max_size 50 -> 10)`.

**The causal chain, timestamp by timestamp:**
- 14:00–14:15 — healthy, 62–65 active connections, 0 waiting, 31–33% utilization.
- 14:20 — 71 active, 3 waiting: "Slight uptick in queued acquisitions from payment-service clients."
- 14:23 — 88 active, 19 waiting: pool "saturated, requests queuing on client side."
- 14:25 — 160 active, 77 waiting: clients "opening redundant retry connections."
- 14:40 (now) — 198/200 active, 143 waiting, 99% utilization, "no mitigation applied yet."

**Ruled out alternatives:**
- **Traffic spike** (the root cause of the very similar past incident
  INC-2024-0417, flash sale at 14:02): *not supported here.*
  Payment-service request cadence in the logs is flat from 14:15 onward —
  no volume increase. DB active connections were steady right up to the
  14:15 deploy. The rise starts at the deploy, not before it.
- **Other deploys:** order-service's only deploy is 2 days earlier;
  database's was a routine patch 8+ hours before onset. Neither correlates.
- **Auth/Kafka:** both healthy — not involved.

## Confidence

**High.** The deploy description, the payment-service log line, the error
message literally quoting `max_size=10`, and the connection-pool timeline
all independently converge on the same 14:15 config change.

## Suggested next steps

1. Roll back `dep-1045` / restore `db_pool.max_size` to 50.
2. Don't just restart payment-service — INC-2024-0417's lesson applies:
   restarts fix leaked connections, not undersized pools.
3. Consider briefly throttling order-service retries while the DB drains
   the 143-request backlog.
4. Follow up on PERF-881 with a pool size validated against actual
   concurrency, not a flat 5x reduction.

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Build the fake world's search index (only needed once, or after editing docs/)
python build_index.py

# CLI, full trace printed to the terminal
export ANTHROPIC_API_KEY=sk-ant-...
python run_agent.py "Payments started failing around 2:30 PM. What could be the cause?"

# Streamlit UI (also loads ANTHROPIC_API_KEY from a local .env if present,
# or you can paste a key directly into the page)
streamlit run app.py

python test_tools.py       # all 6 MCP tools, real protocol, no LLM involved
python test_retrieval.py   # RAG retrieval quality, graded query by query