Sentinel
by fncreator22
README.md
# Sentinel
**A three-stage guardrail agent for LLM-powered coding assistants.**
Sentinel sits between an LLM agent and its execution environment, reviewing every proposed action before it runs. It integrates with tools like Claude Code, Cursor, and CodeX via the Model Context Protocol (MCP), acting as an always-on safety layer that can block destructive commands, flag scope creep, and maintain a full audit trail of every decision.
Supports **both Stdio (local process) and SSE (web endpoint)** MCP transports for maximum compatibility.
---
## š¬ Demo
---
## The Problem
Autonomous LLM coding agents can execute shell commands, modify files, push to remote repositories, and make network requests. This power comes with real risk: a single poorly-scoped prompt or a hallucinated action can cause data loss, expose credentials, or make irreversible changes to a production system.
Existing solutions are binary ā either the agent runs everything without review, or a human must manually approve every step. Neither scales.
---
## The Solution
Sentinel implements a **multi-stage decision pipeline** that handles the full spectrum from obviously safe to dangerously risky actions, using the fastest and most appropriate tool at each stage:
**Stage 1 ā Rules Engine**: Deterministic pattern matching on a configurable YAML ruleset (`config/rules.yaml`). Handles unambiguous cases (recursive deletes, credential exposure, root-level writes) in microseconds with zero network dependency. Features **mtime hot-reloading** across all processes so rules updated in the dashboard take effect immediately in running MCP servers.
**Stage 2 ā Trained Classifier**: A TF-IDF vectorizer and Logistic Regression classifier trained on a labeled dataset of agent actions. Runs in milliseconds, entirely offline, and produces an explainable risk score with confidence bounds.
**Stage 3 ā LLM Reviewer**: For ambiguous actions that statistical models cannot resolve reliably, a large language model evaluates the action in the context of the user's stated task. This is the only stage that makes a network call, and it only activates when the previous stages are uncertain. Supports Ollama (local), OpenAI, Anthropic, and Google Gemini.
---
## Architecture
```
MCP Client (Claude Code / Cursor / CodeX / Web UI)
|
+-- stdio transport (mcp_server/server.py)
+-- SSE transport (mcp_server/sse_server.py)
|
v
api/main.py (FastAPI) <- REST API, static /dashboard, audit log, config management
|
v
sentinel_core/orchestrator.py
|
+-- Stage 1: sentinel_core/rules_engine.py (config/rules.yaml - mtime hot-reload)
+-- Stage 2: sentinel_core/classifier.py (model_artifacts/model.pkl)
+-- Stage 3: sentinel_core/llm_reviewer.py (sentinel_core/model_manager.py)
|
v
sentinel.db (SQLite) <- append-only audit log
```
---
## Classifier Performance
| Metric | Value |
|---|---|
| Training Examples | 828 (hand-labeled + synthetically generated) |
| Class Balance | 58% safe / 42% risky |
| 5-Fold Cross-Validation Accuracy | 76.3% Ā± 3.0% |
| CV Macro-F1 Score | 75.9% Ā± 3.1% |
| Hold-out Test Accuracy | 75.2% |
| Risky Class Precision | 70% |
| Safe Class Precision | 79% |
| Confident Predictions (Stage 2 handles directly) | 40% of traffic |
| Stage 3 LLM escalation rate | 60% of traffic |
The classifier's confidence threshold is set at 80%. Predictions above this threshold are resolved by Stage 2 without invoking the Stage 3 LLM, reducing average latency and eliminating API cost for 40% of all reviewed actions.
**Top 5 features associated with risky actions:** `bash`, `delete`, `iptables`, `exec`, `secret`
**Top 5 features associated with safe actions:** `version`, `list`, `describe`, `test`, `check`
---
## Project Structure
```
sentinel/
āāā api/
ā āāā main.py FastAPI application, all HTTP endpoints
āāā sentinel_core/
ā āāā orchestrator.py Three-stage pipeline coordinator
ā āāā rules_engine.py Stage 1: YAML rule matching
ā āāā classifier.py Stage 2: sklearn inference
ā āāā llm_reviewer.py Stage 3: LLM reasoning
ā āāā model_manager.py Provider abstraction (Ollama / OpenAI / Anthropic / Gemini)
ā āāā audit_log.py SQLite decision logger
ā āāā model_artifacts/ model.pkl + vectorizer.pkl (gitignored)
āāā mcp_server/
ā āāā server.py MCP stdio server
ā āāā sse_server.py MCP SSE server (port 8002)
āāā dashboard/
ā āāā index.html Single-page control panel
ā āāā app.js Dashboard logic
ā āāā style.css Dashboard styles
āāā config/
ā āāā rules.yaml Stage 1 allow/block patterns
ā āāā model_config.yaml Active provider and model selection
ā āāā model_config.local.yaml API keys (gitignored, never committed)
āāā data/
ā āāā training_examples.csv Labeled dataset for Stage 2 training
āāā train/
ā āāā train_classifier.py Training script (scikit-learn)
ā āāā generate_training_data.py Synthetic training data generation
āāā docs/
ā āāā ARCHITECTURE.md Internal design notes and rationale
āāā start.bat Windows one-click launcher
āāā Dockerfile Container image definition
āāā requirements.txt
```
---
## Design Decisions
### Why three stages instead of one?
The design goal was to minimize latency and cost for the common case while preserving high-accuracy judgment for the ambiguous case. The vast majority of agent actions are either obviously safe (`git status`, `npm install`) or obviously risky (`rm -rf /`, `git push --force`). Routing both through an LLM would be slow and expensive. Routing both through a rules engine alone would miss the large middle ground.
The three-stage cascade solves this:
- **Stage 1** handles the clear-cut cases deterministically, in microseconds, with no model in the loop. A pattern match on a known-dangerous string cannot hallucinate. This is the last line of defense for catastrophic commands.
- **Stage 2** handles the statistical middle ground offline, in milliseconds, with an explainable coefficient-based model. We chose TF-IDF + Logistic Regression deliberately: the model trains in seconds on a CPU, produces inspectable coefficients, and is well-suited to short action text where risk is concentrated in specific keywords and n-grams. A neural network would add opacity without meaningfully improving the problem.
- **Stage 3** handles genuine ambiguity ā cases where context (the user's stated task, the scope of the session) matters more than surface-level tokens. This is where an LLM's reasoning ability adds real value, and it is the only stage that pays the latency and cost of a model call.
### Why local-first for Stage 3?
We implemented Stage 3 with Ollama as the default to ensure that no action text leaves the user's machine unless they explicitly configure a cloud provider. This is important for codebases that may contain proprietary logic, internal hostnames, or sensitive file paths. The provider abstraction in `model_manager.py` makes it straightforward to switch to a cloud LLM without changing any Stage 3 logic.
### Why a confidence threshold?
Stage 2 does not pass every prediction to Stage 3 ā only predictions below an 80% confidence threshold. This gates the expensive network call behind a statistical signal. Predictions above the threshold are resolved by Stage 2 directly, which accounts for approximately 40% of all traffic in practice. The remaining 60% escalates to Stage 3, where LLM reasoning provides the most marginal value.
---
---
## Setup & Quick Start
### Step 1: Start Sentinel Backend & Dashboard
**Windows (One-Click)**:
Double-click `start.bat` in the project root folder. The script will automatically:
1. Create a Python virtual environment (`venv`) if missing
2. Install required packages from `requirements.txt`
3. Train the Stage 2 ML classifier if model pickles are missing.
4. Launch the FastAPI backend on `http://localhost:8000`
5. Launch the SSE server on `http://localhost:8002`
6. Launch the Live Dashboard on `http://localhost:8080` in your default browser.
**Manual / Linux / macOS Setup**:
```bash
# 1. Create and activate virtual environment
python -m venv venv
source venv/bin/activate # macOS/Linux (use venv\Scripts\activate on Windows)
# 2. Install dependencies
pip install -r requirements.txt
# 3. Train classifier model (first run only)
python train/train_classifier.py
# 4. Run API Server (Terminal 1)
python -m uvicorn api.main:app --port 8000 --reload
# 5. Run SSE MCP Server (Terminal 2)
python mcp_server/sse_server.py --port 8002
# 6. Run Dashboard UI (Terminal 3)
python -m http.server 8080 --directory dashboard
```
---
## Step-by-Step Client Integration Guide
Sentinel seamlessly connects to any MCP-compliant AI assistant. Follow the exact step-by-step guide below for your platform:
### 1. Claude Desktop (Windows / macOS)
1. **Start Sentinel**: Ensure `start.bat` or the backend services are running.
2. **Open Configuration File**:
- **Windows**: Open `%APPDATA%\Claude\claude_desktop_config.json` in Notepad or VS Code.
- **macOS**: Open `~/Library/Application Support/Claude/claude_desktop_config.json`.
3. **Paste the Configuration**:
Add `sentinel` under `mcpServers` with the absolute path to your Python virtual environment executable and `mcp_server/server.py`:
```json
{
"mcpServers": {
"sentinel": {
"command": "C:\\path\\to\\sentinel\\venv\\Scripts\\python.exe",
"args": [
"C:\\path\\to\\sentinel\\mcp_server\\server.py"
],
"env": {
"PYTHONPATH": "C:\\path\\to\\sentinel"
}
}
}
}
```
4. **Restart Claude Desktop**: Completely close and relaunch Claude Desktop.
5. **Verify Connection**:
- In Claude Desktop, click the **Hammer šØ / Settings** icon in the bottom right corner of the chat window, or go to **Settings > Developer**.
- You will see a blue badge reading **`sentinel running`** with active tools `review_action` and `get_recent_decisions`.
---
### 2. Cursor IDE (Stdio & SSE Transport)
1. **Open Cursor Settings**: Open Cursor IDE, click **Settings (Gear Icon)** in the top right or press `Ctrl + ,` / `Cmd + ,`.
2. **Navigate to MCP**: Select **Features** from the sidebar, then scroll down to **MCP Servers**.
3. **Add New Server**:
- Click **+ Add New MCP Server**.
- **Name**: `sentinel`
- **Type**: Select `SSE` (recommended for zero sub-process overhead) or `stdio`.
- **URL / Command**:
- For **SSE**: Enter `http://localhost:8002/sse`.
- For **stdio**: Set `command` to your `python.exe` and `args` to `mcp_server/server.py`.
4. **Verify**: The status indicator will turn **Green (Connected)**.
---
### 3. Claude Code CLI
1. **Locate Config**: Open `~/.claude/claude_code_config.json` (or project-level `.claude/config.json`).
2. **Add MCP Server**:
```json
{
"mcpServers": {
"sentinel": {
"command": "python",
"args": ["mcp_server/server.py"],
"cwd": "/path/to/sentinel"
}
}
}
```
3. **Run Prompt**: When Claude Code proposes commands, it will invoke `review_action` automatically before executing.
---
### 4. Web-Based IDEs & Custom HTTP Clients (SSE)
For web platforms, remote agents, or testing with the **MCP Inspector**:
- **SSE Endpoint**: `http://localhost:8002/sse`
- **Messages Endpoint**: `http://localhost:8002/messages/`
- **Test with Inspector**:
```bash
npx -y @modelcontextprotocol/inspector sse http://localhost:8002/sse
```
---
### 5. Rapid Connection Utility & Real-Time Syncing Optimizer in Dashboard
The dashboard provides a built-in interactive server connection manager and real-time syncing optimizer:
1. **Live Syncing Status**: The header badge **`LIVE SYNC (xx ms)`** dynamically monitors API latency, continuously updating decision logs, active review progress bars, and model health without manual page reloads.
2. **Cloud & Remote Connection**: Click **Connect** or the **LIVE SYNC** badge to configure custom backend API URLs, SSE transport endpoints, or API keys for remote cloud deployments.
3. **Auto-Generated Configs**: Switch between **Server Sync**, **Stdio**, and **SSE** tabs to copy auto-filled MCP configuration snippets customized to your local file paths.
4. Click **Copy Config to Clipboard** and paste directly into your client's config file!
---
## Configuration
### Stage 3 Model Provider
Open the dashboard at `http://localhost:8080` and navigate to **Model Settings**.
**Local (Ollama)**: Select any model detected from your local Ollama installation. No internet required. Pull a model with `ollama pull <model-name>` before selecting it.
**API providers**: Select Google Gemini, OpenAI, Anthropic, or a custom OpenAI-compatible endpoint. Enter your API key and click **Save Model Settings**. After saving, the dashboard automatically calls the provider's live `/models` endpoint and populates the model dropdown with every model your key has access to. You can then pick from the list or type any model name manually. If you leave the model field blank, Sentinel auto-selects the recommended default for that provider. The key is stored in `config/model_config.local.yaml` on disk, never written to the audit log, and never returned in full over the API (only the last 4 characters are shown in the dashboard).
Click **ā» Refresh** next to the model field at any time to re-fetch the live model list without saving again.
### Stage 1 Rules
Navigate to the **Rules** tab in the dashboard to add, edit, or remove pattern-matching rules. Rules support exact substring matching and regular expressions. Changes take effect immediately without restarting the server.
---
## MCP Integration & Tools Initialization
Sentinel exposes its capabilities to coding assistants via the Model Context Protocol (MCP). The server uses `FastMCP` to initialize the tools and handle the transport layer (both `stdio` and `sse` are supported).
### Tools Initialization
When the MCP server starts, it initializes the following tools and makes them available to the connected client:
| Tool | Initialization & Arguments | Description |
|---|---|---|
| `review_action` | `action_text` (str), `user_task` (str) | Review a proposed agent action BEFORE executing it. Returns a verdict of `ALLOW`, `BLOCK`, or `REVIEW`. |
| `get_recent_decisions` | `limit` (int, default=20) | Return recent entries from the audit log to provide the agent with context of past verdicts. |
---
## API Reference
The FastAPI backend exposes the following endpoints. Full interactive documentation is available at `http://localhost:8000/docs` when the server is running.
| Method | Path | Description |
|---|---|---|
| GET | `/health` | Health check, returns paused state and project path |
| GET | `/status` | Lightweight paused/auth status |
| POST | `/review` | Submit an action for review |
| POST | `/pause` | Pause the guardrail pipeline |
| POST | `/resume` | Resume the guardrail pipeline |
| GET | `/log` | Retrieve recent audit log entries |
| GET | `/rules` | Get current Stage 1 ruleset |
| POST | `/rules` | Update Stage 1 ruleset |
| GET | `/models/local` | List locally available Ollama models |
| GET | `/models/config` | Get current model provider configuration |
| POST | `/models/config` | Update model provider configuration |
| POST | `/models/test` | Test the active model provider connection |
| GET | `/models/available` | Fetch live list of models available on the saved API key |
| GET | `/mcp/servers` | List registered MCP servers |
| POST | `/mcp/servers` | Register a new MCP server |
| DELETE | `/mcp/servers/{name}` | Remove a registered MCP server |
### Example: Reviewing an action
```bash
curl -X POST http://localhost:8000/review \
-H "Content-Type: application/json" \
-d '{"action_text": "rm -rf /tmp/build", "user_task": "Clean up build artifacts"}'
```
Response:
```json
{
"action_text": "rm -rf /tmp/build",
"verdict": "ALLOW",
"decided_by_stage": "classifier",
"reason": "Classifier predicted 'safe' with 89% confidence.",
"log_id": 42
}
```
---
## Stage 2 Classifier ā Training
The classifier is trained on `data/training_examples.csv`, a hand-curated and synthetically augmented dataset of shell commands, SQL statements, git operations, and API calls labeled as `safe` or `risky`.
To generate additional synthetic training data:
```bash
python train/generate_training_data.py
```
To retrain after modifying or expanding the dataset:
```bash
python train/train_classifier.py
```
The script prints a full classification report and the top 15 features associated with risk, allowing the model's learned signals to be verified and understood without treating it as a black box.
---
## Security Notes
- API keys are stored only in `config/model_config.local.yaml`, which is gitignored by default.
- The audit log (`sentinel.db`) records action text and review decisions but never stores API keys.
- If `SENTINEL_API_KEY` is set as an environment variable before starting the server, all mutating endpoints (rules update, model config update, pause/resume) require that key via the `X-Sentinel-Key` header.
- The guardrail can be paused from the dashboard. In paused mode, all actions return `REVIEW`, requiring manual sign-off. This is intentionally conservative.
---
## License
MIT License. See `LICENSE` for details.
This server cannot be deployed
Maintenance
ActivitySlowing
ResponsivenessNo issues