Sentinel MCP

Production Incident Investigation & AI Software Engineering Evaluation Platform

Sentinel MCP is a distributed e-commerce backend and benchmarking platform designed to evaluate human software engineers and autonomous AI coding agents on realistic distributed system incidents using standard Model Context Protocol (MCP) diagnostic tools.

Architecture Overview

flowchart TB
    subgraph Clients
        Agent[AI Agent / Engineer]
        User[End User / Load Gen]
    end

    subgraph "MCP Engineering Gateway"
        MCP_Repo[Repository MCP]
        MCP_Logs[Logs MCP]
        MCP_Metrics[Metrics MCP]
        MCP_Traces[Traces MCP]
        MCP_DB[Database MCP]
        MCP_Deploy[Deployment MCP]
    end

    subgraph "Application Services"
        GW[API Gateway :8000]
        Orders[Order Service :8001]
        Inventory[Inventory Service :8002]
        Payment[Payment Service :8003]
        Notifications[Notification Worker :8004]
        OutboxWorker[Outbox Worker]
        IncidentCP[Incident Control Plane :8005]
    end

    subgraph "Infrastructure & Storage"
        DB[(PostgreSQL 16)]
        Redis[(Redis 7)]
        Kafka[(Redpanda / Kafka)]
        OTel[OpenTelemetry Collector]
        Prom[Prometheus]
        Grafana[Grafana]
    end

    Agent <-->|Official MCP Protocol| MCP_Repo & MCP_Logs & MCP_Metrics & MCP_Traces & MCP_DB & MCP_Deploy
    User --> GW
    GW --> Orders
    Orders --> Inventory
    Orders --> Payment
    Orders -->|Atomic Tx| DB
    Orders -->|Outbox Table| DB
    OutboxWorker -->|Poll SKIP LOCKED| DB
    OutboxWorker -->|Publish| Kafka
    Kafka --> Notifications
    IncidentCP -->|Fault Injection| Orders & Inventory & Payment & GW
    Orders & Inventory & Payment -.->|Metrics/Traces| OTel
    OTel --> Prom
    Prom --> Grafana

Related MCP server: AIOps MCP

Core Engineering Features

1. Explicit Domain State Machine:

   • Order lifecycle strictly governed by legal transitions (PENDING -> INVENTORY_RESERVED -> PAYMENT_PENDING -> PAID -> PROCESSING -> COMPLETED).

   • Domain invariants encapsulated within aggregate methods, raising InvalidStateTransitionError on illegal mutations.

2. Concurrency & Zero Oversell Guarantee:

   • Multi-item inventory reservations acquire row locks in deterministic sorted order (product_id) to eliminate database deadlocks.

   • High-concurrency verified: 50 concurrent checkouts competing for 10 units of inventory yield exactly 10 successes and 40 conflict rejections.

3. Deterministic Payment Simulator & Idempotency:

   • Zero real card exposure; uses SHA-256 body fingerprinting to enforce distributed idempotency.

   • Supports configurable simulated faults: timeouts, bank declines, 502/500 gateway errors, and network jitter.

4. Transactional Outbox & Exactly-Once Semantics:

   • Orders and events written atomically inside single database transactions.

   • Background OutboxWorker manages leases, retries with exponential backoff and jitter, and dead-letter queues (DLQ).

   • Event consumers utilize an atomic processed_events store to ensure at-least-once deliveries produce exactly-once business side effects.

5. Observability Stack:

   • Structured JSON logging with W3C traceparent and correlation IDs.

   • OpenTelemetry distributed tracing across HTTP, Kafka, and database layers.

   • Prometheus metrics and pre-built Grafana dashboards (sentinel_overview.json, kafka_outbox_pipeline.json, incident_investigation.json).

6. Model Context Protocol (MCP) Diagnostic Gateway:

   • Implements official MCP specification (mcp 1.26.0).

   • 6 tool categories: Repository, Logs, Metrics, Traces, Database, and Deployment.

   • Secure sandboxing: blocks path traversal (../), credentials, .env, golden.patch, and arbitrary shell execution.

Quickstart

1. Local Environment Setup

# Clone the repository
git clone https://github.com/sentinel-mcp/sentinel-mcp.git
cd sentinel-mcp

# Install dependencies
pip install -e ".[dev,db,kafka]"

# Run full test suite
pytest tests/ -v

2. Docker Compose Environment

# Start all microservices, Redpanda, PostgreSQL, Prometheus, Grafana
docker-compose up -d

# Inspect health endpoints
curl http://localhost:8000/health
curl http://localhost:8005/incidents

15 Reproducible Incident Scenarios

Deterministic Evaluation Engine

The evaluation platform assesses candidate solutions with zero LLM judge bias:

# Run deterministic evaluator for a scenario
python sentinel_eval.py --scenario 01_payment_timeout_regression --candidate my_agent

Scoring Matrix

Functional Correctness      45%
Hidden Tests & Invariants   20%
Regression Protection       10%
Performance SLA             10%
Security & Sandbox Checks   10%
Required MCP Discovery       5%
--------------------------------
Total:                     100%

The evaluator produces a detailed machine-readable report in evaluation_report.json.

Investigating Incidents with MCP Tools

When an incident is active, agents can inspect the system using safe diagnostic tools:

# 1. Check service error rates
get_error_rate(service="orders")

# 2. Inspect recent errors
get_service_errors(service="orders", limit=10)

# 3. Locate slow distributed transactions
search_slow_traces(min_duration_ms=100.0)

# 4. Compare deployment configurations
compare_configuration(deployment_id_a="dep-v2.4.1", deployment_id_b="dep-v2.4.2")

# 5. Check database query execution plan
explain_query("SELECT * FROM orders WHERE customer_id = 'cust_123'")

Architecture Decision Records (ADRs)

Detailed architectural justifications are located in docs/adr/:

• ADR-001: Why Event-Driven Architecture

• ADR-002: Why PostgreSQL

• ADR-003: Why Transactional Outbox

• ADR-004: Kafka / Redpanda vs Alternatives

• ADR-005: MCP Specification & Security Model

• ADR-006: Fault-Injection Architecture

• ADR-007: Idempotency Strategy

License

Apache-2.0 License. See LICENSE for details.