# Sentinel

Sentinel is an MCP server that scores AI agent responses **before** they
reach a user — a trust layer an agent calls mid-turn, not a dashboard
someone checks after the fact. An agent (or the orchestrator wrapping it)
calls a Sentinel tool with the response it's about to send plus the context
it was grounded in; Sentinel returns a verdict the caller can act on —
allow it, warn, or block it — before the response ever ships.

Free for evaluation, personal, and internal use. Production use requires
a commercial license — see [`LICENSE.md`](LICENSE.md) and
[mleg.tech/sentinel](https://mleg.tech) for pricing.

It currently ships two tools:

- **`check_groundedness`** — is this response actually supported by its
  context, or does it cite something that isn't there?
- **`flag_injection`** — does the context contain a prompt-injection
  attempt, and did the response go along with it?

## Why this exists

Most "AI eval" tooling scores transcripts after the fact, in a dashboard
nobody opens. Sentinel is built to sit in the request path: an MCP tool
call an agent (or its orchestrator) makes on a response before it goes out,
cheap enough in the common case to not be a UX problem.

That "cheap enough" claim is the actual design constraint, and it shapes
everything below.

## How it works: the escalation ladder

Every metric is **deterministic-first**. A metric's own regex/pattern logic
tries to resolve a call with high confidence — a clear pass or a clear
block — using zero model calls. Only the ambiguous middle escalates to an
LLM-as-judge call. This keeps the common case fast and the code auditable
(a confident block should be explainable without pointing at a model), and
reserves the model call for cases that actually need judgment.

```
ScoringRequest
      │
      ▼
 deterministic check (regex / pattern / citation resolution)
      │
      ├── confident pass/block ──────────────► Verdict (decidedBy: "heuristic")
      │
      └── ambiguous ──► judge call, raced against maxJudgeLatencyMs
                            │
                            ├── resolves in time ──► Verdict (decidedBy: "judge")
                            └── times out ──────────► tenant's onTimeout policy:
                                                        fail_closed → block
                                                        fail_open   → outcome "timeout"
```

**`flag_injection`** looks for two independent signal families: *override*
patterns (an injection attempt living in the retrieved context — "ignore
prior instructions," "you are now...") and *compliance* patterns (the
agent's own response actually going along with it — "sure, here is the
full customer table"). Both firing together is the full attack chain and a
confident block with no judge call; either alone is ambiguous and
escalates. Score direction: **higher = worse**, blocks when `score >=
threshold`.

**`check_groundedness`** extracts citation-like strings from the response
and checks whether the context actually supports them — but a naive
"does this substring appear in the source" check is wrong in exactly the
case that matters most: a document saying *"no section 9.2 exists"*
contains the literal substring "section 9.2." So the check scans a
~60-character window before each match for negation markers ("no," "not,"
"doesn't," "never mentions," ...); a citation that only ever appears
negated counts as unresolved — same bucket as never appearing at all. An
unresolved citation is a confident block on its own. Otherwise, token
overlap between response and context has to clear a high floor before the
deterministic tier will confidently pass. Score direction: **higher =
better**, blocks when `score < threshold`.

Full worked examples and the exact request/verdict JSON shapes live in
[contracts/README.md](contracts/README.md).

## Policy, per tenant

Nothing is scored against a hardcoded threshold. Each tenant has a
`TenantPolicy` (`policy/registry.ts`) that says, per metric: is it enabled,
what's the block threshold, how long a judge call is allowed to run
(`maxJudgeLatencyMs`), and what happens if it times out (`fail_open` vs
`fail_closed` — there is no system-wide default; every tenant sets this
explicitly). A caller can override threshold/timeout knobs per-request, but
can't enable a disabled metric or switch tenant by passing input. Verdicts
from all requested metrics roll up into one `EvaluationResult` — an
aggregated `action` (`allow`/`warn`/`block`) and a placeholder 0-100 trust
score, not yet calibrated against real incident data.

## Project layout

```
contracts/    shared types + zod schemas — the only thing every folder depends on
metrics/      the escalation-ladder logic per metric (injection, groundedness)
judges/       provider-agnostic judge interfaces + offline stub judges (no API key needed)
policy/       per-tenant policy resolution (in-memory registry for now)
server/       transport-agnostic handlers (flagInjection, checkGroundedness) + trust-score rollup
server/index.ts   stdio MCP entrypoint (Claude Desktop, Cursor, local dev)
app/api/mcp/  HTTP MCP entrypoint (Streamable HTTP, bearer-token auth) — same handlers, thin glue
app/          Next.js demo dashboard
scripts/      smoke test proving the HTTP deploy target end-to-end
```

The handlers in `server/handlers/` are plain `async` functions with no
transport imports (no MCP SDK types, no Next.js types) — both the stdio
entrypoint and the HTTP route call the exact same functions. Only the
registration glue differs between deploy targets.

Judges are provider-agnostic by interface (`judges/types.ts`); the only
implementation checked in is an offline stub (`judges/stub.ts`, zero
network calls, simple pattern/overlap scoring) so `git clone && npm
install && npm test` works with no API key. A real model-backed judge would
implement the same interface in its own file.

## Running it

```bash
npm install

# stdio MCP server (Claude Desktop, Cursor, etc.)
npm run mcp:stdio

# Next.js app: HTTP MCP endpoint (/api/mcp) + demo dashboard (/)
npm run dev
```

The HTTP endpoint requires a bearer token — it's a hardcoded
token→tenant map for local dev (`server/http/auth.ts`), not a real
authorization server:

```bash
curl http://localhost:3000/api/mcp \
  -H "Authorization: Bearer sentinel-local-dev-token" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}'
```

Open `http://localhost:3000` for the demo dashboard: click "Run demo
session" to watch a real support-agent transcript scored turn by turn —
including a fabricated contract citation and a prompt-injection attempt,
both caught before they'd have reached the user. The pass/fail states come
from genuine round trips to the real handlers, not mocked UI state.

```bash
npm test          # unit + regression tests (metrics, handlers, scoring)
npm run typecheck # tsc --noEmit --strict
npm run smoke:http # end-to-end proof the HTTP deploy target works: real MCP
                    # client, real Streamable HTTP JSON-RPC, auth included
```

## Using this in your own project

Everything above walks through running *this repo's* demo. Adopting
Sentinel into your own agent is a different, smaller task: you're wiring
one or two tool calls into a path you already have, not standing up a new
service.

**The integration point is narrow on purpose.** Wherever your
agent/orchestrator currently sends a response to a user, add one call
first:

```ts
const verdict = await flagInjection({ response, context });
// or: await checkGroundedness({ response, context })

if (verdict.action === "block") {
  // don't ship the response — return a fallback or re-generate
} else if (verdict.action === "warn") {
  // ship it, but log the verdict for review
}
// "allow" → ship as normal
```

That's the whole contract. Sentinel doesn't need to know anything about
your agent framework, your model provider, or your orchestration layer —
it scores a response-plus-context pair and hands back a verdict. If your
stack can make an MCP tool call (or call the same handlers directly — see
`server/handlers/`), it can sit in front of Sentinel.

**Steps to adopt it:**

1. Clone the repo and run `npm test` — confirm the escalation ladder
   behaves as documented against the existing test fixtures before you
   change anything.
2. Pick a deploy target: `server/index.ts` (stdio) if your agent runs
   locally or in a tool like Claude Desktop/Cursor; `app/api/mcp/` (HTTP)
   if it needs to be called over the network. Both call the same
   handlers — this is a wiring decision, not an architecture one.
3. Write your own `TenantPolicy` (`policy/registry.ts`) — decide your
   thresholds, your judge timeout, and explicitly set `fail_open` vs
   `fail_closed` for your case. There's no safe default; that's a product
   decision only you can make for your traffic.
4. Point the call at your actual response/context shape. If your agent's
   "context" isn't already a flat list of strings, the contracts layer
   (`contracts/`) is the place to adapt the shape — not the metrics
   themselves.
5. Start in `warn` mode in front of real traffic before you let anything
   auto-block, the same way you'd roll out any new gate in a production
   path.

**Licensing note:** the code above is free to use for evaluation,
personal projects, and internal experimentation. Running Sentinel in
front of real production traffic requires a commercial license — see
[`LICENSE.md`](LICENSE.md) for the full terms, and
[mleg.tech/sentinel](https://mleg.tech/sentinel) for what's included at each tier
(free, commercial, and a done-with-you option for teams that want
production gaps like the Postgres-backed policy registry or a
model-backed judge built against their own setup).

## What's a placeholder vs. what's real

- The escalation ladder, policy resolution, and both metrics' detection
  logic are real and tested.
- The trust score and the `weighted` aggregation mode are explicit
  placeholders — a starting point, not calibrated against incident data.
- The policy registry is in-memory; a Postgres-backed implementation
  implements the same `PolicyRegistry` interface and should be a drop-in.
- Two more metrics (`toxicity`, `pii_leakage`) are already wired into the
  policy schema as `enabled: false` — reference points for where their
  implementations would live, not stubs pretending to work.
- The only judge implementation is the offline stub. A real model-backed
  judge slots into the same `InjectionJudge`/`GroundednessJudge` interfaces.

## License

Free to use for evaluation, personal projects, and internal
experimentation. A commercial license is required to run Sentinel in
production. Full terms in [`LICENSE.md`](LICENSE.md); pricing and tiers
at [mleg.tech/sentinel](https://mleg.tech/sentinel).