# Sentinel MCP

[![CI/CD](https://github.com/sentinel-mcp/sentinel-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/sentinel-mcp/sentinel-mcp/actions)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org/)
[![MCP](https://img.shields.io/badge/mcp-1.26.0-green.svg)](https://modelcontextprotocol.io/)

> **Production Incident Investigation & AI Software Engineering Evaluation Platform**

Sentinel MCP is a distributed e-commerce backend and benchmarking platform designed to evaluate human software engineers and autonomous AI coding agents on realistic distributed system incidents using standard **Model Context Protocol (MCP)** diagnostic tools.

---

## Architecture Overview

```mermaid
flowchart TB
    subgraph Clients
        Agent[AI Agent / Engineer]
        User[End User / Load Gen]
    end

    subgraph "MCP Engineering Gateway"
        MCP_Repo[Repository MCP]
        MCP_Logs[Logs MCP]
        MCP_Metrics[Metrics MCP]
        MCP_Traces[Traces MCP]
        MCP_DB[Database MCP]
        MCP_Deploy[Deployment MCP]
    end

    subgraph "Application Services"
        GW[API Gateway :8000]
        Orders[Order Service :8001]
        Inventory[Inventory Service :8002]
        Payment[Payment Service :8003]
        Notifications[Notification Worker :8004]
        OutboxWorker[Outbox Worker]
        IncidentCP[Incident Control Plane :8005]
    end

    subgraph "Infrastructure & Storage"
        DB[(PostgreSQL 16)]
        Redis[(Redis 7)]
        Kafka[(Redpanda / Kafka)]
        OTel[OpenTelemetry Collector]
        Prom[Prometheus]
        Grafana[Grafana]
    end

    Agent <-->|Official MCP Protocol| MCP_Repo & MCP_Logs & MCP_Metrics & MCP_Traces & MCP_DB & MCP_Deploy
    User --> GW
    GW --> Orders
    Orders --> Inventory
    Orders --> Payment
    Orders -->|Atomic Tx| DB
    Orders -->|Outbox Table| DB
    OutboxWorker -->|Poll SKIP LOCKED| DB
    OutboxWorker -->|Publish| Kafka
    Kafka --> Notifications
    IncidentCP -->|Fault Injection| Orders & Inventory & Payment & GW
    Orders & Inventory & Payment -.->|Metrics/Traces| OTel
    OTel --> Prom
    Prom --> Grafana
```

---

## Core Engineering Features

1. **Explicit Domain State Machine**:
   - Order lifecycle strictly governed by legal transitions (`PENDING -> INVENTORY_RESERVED -> PAYMENT_PENDING -> PAID -> PROCESSING -> COMPLETED`).
   - Domain invariants encapsulated within aggregate methods, raising `InvalidStateTransitionError` on illegal mutations.
2. **Concurrency & Zero Oversell Guarantee**:
   - Multi-item inventory reservations acquire row locks in deterministic sorted order (`product_id`) to eliminate database deadlocks.
   - High-concurrency verified: 50 concurrent checkouts competing for 10 units of inventory yield exactly 10 successes and 40 conflict rejections.
3. **Deterministic Payment Simulator & Idempotency**:
   - Zero real card exposure; uses SHA-256 body fingerprinting to enforce distributed idempotency.
   - Supports configurable simulated faults: timeouts, bank declines, 502/500 gateway errors, and network jitter.
4. **Transactional Outbox & Exactly-Once Semantics**:
   - Orders and events written atomically inside single database transactions.
   - Background `OutboxWorker` manages leases, retries with exponential backoff and jitter, and dead-letter queues (DLQ).
   - Event consumers utilize an atomic `processed_events` store to ensure at-least-once deliveries produce exactly-once business side effects.
5. **Observability Stack**:
   - Structured JSON logging with W3C `traceparent` and correlation IDs.
   - OpenTelemetry distributed tracing across HTTP, Kafka, and database layers.
   - Prometheus metrics and pre-built Grafana dashboards (`sentinel_overview.json`, `kafka_outbox_pipeline.json`, `incident_investigation.json`).
6. **Model Context Protocol (MCP) Diagnostic Gateway**:
   - Implements official MCP specification (`mcp 1.26.0`).
   - 6 tool categories: Repository, Logs, Metrics, Traces, Database, and Deployment.
   - Secure sandboxing: blocks path traversal (`../`), credentials, `.env`, `golden.patch`, and arbitrary shell execution.

---

## Quickstart

### 1. Local Environment Setup

```bash
# Clone the repository
git clone https://github.com/sentinel-mcp/sentinel-mcp.git
cd sentinel-mcp

# Install dependencies
pip install -e ".[dev,db,kafka]"

# Run full test suite
pytest tests/ -v
```

### 2. Docker Compose Environment

```bash
# Start all microservices, Redpanda, PostgreSQL, Prometheus, Grafana
docker-compose up -d

# Inspect health endpoints
curl http://localhost:8000/health
curl http://localhost:8005/incidents
```

---

## 15 Reproducible Incident Scenarios

| Incident | Category | Title & Root Cause |
|---|---|---|
| `01_payment_timeout_regression` | Performance | Deployment reduced payment timeout from 2.0s to 0.05s while provider takes 0.12s |
| `02_db_connection_leak` | Resource Leak | Async DB session in checkout audit path acquired without release; pool exhausted |
| `03_n_plus_one_query` | Database | Order listing query executes individual child queries in a loop |
| `04_cache_stampede` | Caching | Hot cache key expires without request coalescing or mutex lock |
| `05_duplicate_kafka_event` | Messaging | Consumer missing deduplication table executes duplicate notifications |
| `06_poison_event` | Messaging | Malformed message payload causes consumer crash loop without DLQ |
| `07_retry_storm` | Resilience | Gateway and services retry immediately without backoff/jitter |
| `08_dead_worker` | Worker | Worker claims outbox job and crashes before completion; missing lease expiry |
| `09_db_deadlock` | Concurrency | Concurrent reservations lock inventory items in opposite order |
| `10_inventory_race` | Concurrency | Unsynchronized stock check oversells inventory below zero |
| `11_memory_leak` | Resource Leak | Consumer appends all message payloads to unbounded global list |
| `12_missing_db_index` | Database | Orders query on `(customer_id, created_at)` lacks composite index |
| `13_cache_invalidation_failure` | Caching | Stock confirmation updates database but omits Redis cache eviction |
| `14_event_schema_incompatibility` | Messaging | Producer introduces breaking schema field renaming without backwards compatibility |
| `15_deployment_config_regression` | Deployment | Corrupted environment variable `PAYMENT_SERVICE_URL` injected in release |

---

## Deterministic Evaluation Engine

The evaluation platform assesses candidate solutions with zero LLM judge bias:

```bash
# Run deterministic evaluator for a scenario
python sentinel_eval.py --scenario 01_payment_timeout_regression --candidate my_agent
```

### Scoring Matrix

```text
Functional Correctness      45%
Hidden Tests & Invariants   20%
Regression Protection       10%
Performance SLA             10%
Security & Sandbox Checks   10%
Required MCP Discovery       5%
--------------------------------
Total:                     100%
```

The evaluator produces a detailed machine-readable report in `evaluation_report.json`.

---

## Investigating Incidents with MCP Tools

When an incident is active, agents can inspect the system using safe diagnostic tools:

```python
# 1. Check service error rates
get_error_rate(service="orders")

# 2. Inspect recent errors
get_service_errors(service="orders", limit=10)

# 3. Locate slow distributed transactions
search_slow_traces(min_duration_ms=100.0)

# 4. Compare deployment configurations
compare_configuration(deployment_id_a="dep-v2.4.1", deployment_id_b="dep-v2.4.2")

# 5. Check database query execution plan
explain_query("SELECT * FROM orders WHERE customer_id = 'cust_123'")
```

---

## Architecture Decision Records (ADRs)

Detailed architectural justifications are located in `docs/adr/`:
- [ADR-001: Why Event-Driven Architecture](docs/adr/ADR-001-why-event-driven-architecture.md)
- [ADR-002: Why PostgreSQL](docs/adr/ADR-002-why-postgresql.md)
- [ADR-003: Why Transactional Outbox](docs/adr/ADR-003-why-transactional-outbox.md)
- [ADR-004: Kafka / Redpanda vs Alternatives](docs/adr/ADR-004-kafka-vs-alternatives.md)
- [ADR-005: MCP Specification & Security Model](docs/adr/ADR-005-mcp-security-model.md)
- [ADR-006: Fault-Injection Architecture](docs/adr/ADR-006-fault-injection-architecture.md)
- [ADR-007: Idempotency Strategy](docs/adr/ADR-007-idempotency-strategy.md)

---

## License

Apache-2.0 License. See [LICENSE](LICENSE) for details.