Sentinel

Sentinel is an MCP server that scores AI agent responses before they reach a user — a trust layer an agent calls mid-turn, not a dashboard someone checks after the fact. An agent (or the orchestrator wrapping it) calls a Sentinel tool with the response it's about to send plus the context it was grounded in; Sentinel returns a verdict the caller can act on — allow it, warn, or block it — before the response ever ships.

It currently ships two tools:

• check_groundedness — is this response actually supported by its context, or does it cite something that isn't there?

• flag_injection — does the context contain a prompt-injection attempt, and did the response go along with it?

Why this exists

Most "AI eval" tooling scores transcripts after the fact, in a dashboard nobody opens. Sentinel is built to sit in the request path: an MCP tool call an agent (or its orchestrator) makes on a response before it goes out, cheap enough in the common case to not be a UX problem.

That "cheap enough" claim is the actual design constraint, and it shapes everything below.

How it works: the escalation ladder

Every metric is deterministic-first. A metric's own regex/pattern logic tries to resolve a call with high confidence — a clear pass or a clear block — using zero model calls. Only the ambiguous middle escalates to an LLM-as-judge call. This keeps the common case fast and the code auditable (a confident block should be explainable without pointing at a model), and reserves the model call for cases that actually need judgment.

ScoringRequest
      │
      ▼
 deterministic check (regex / pattern / citation resolution)
      │
      ├── confident pass/block ──────────────► Verdict (decidedBy: "heuristic")
      │
      └── ambiguous ──► judge call, raced against maxJudgeLatencyMs
                            │
                            ├── resolves in time ──► Verdict (decidedBy: "judge")
                            └── times out ──────────► tenant's onTimeout policy:
                                                        fail_closed → block
                                                        fail_open   → outcome "timeout"

flag_injection looks for two independent signal families: override patterns (an injection attempt living in the retrieved context — "ignore prior instructions," "you are now...") and compliance patterns (the agent's own response actually going along with it — "sure, here is the full customer table"). Both firing together is the full attack chain and a confident block with no judge call; either alone is ambiguous and escalates. Score direction: higher = worse, blocks when score >= threshold.

check_groundedness extracts citation-like strings from the response and checks whether the context actually supports them — but a naive "does this substring appear in the source" check is wrong in exactly the case that matters most: a document saying "no section 9.2 exists" contains the literal substring "section 9.2." So the check scans a ~60-character window before each match for negation markers ("no," "not," "doesn't," "never mentions," ...); a citation that only ever appears negated counts as unresolved — same bucket as never appearing at all. An unresolved citation is a confident block on its own. Otherwise, token overlap between response and context has to clear a high floor before the deterministic tier will confidently pass. Score direction: higher = better, blocks when score < threshold.

Full worked examples and the exact request/verdict JSON shapes live in contracts/README.md.

Policy, per tenant

Nothing is scored against a hardcoded threshold. Each tenant has a TenantPolicy (policy/registry.ts) that says, per metric: is it enabled, what's the block threshold, how long a judge call is allowed to run (maxJudgeLatencyMs), and what happens if it times out (fail_open vs fail_closed — there is no system-wide default; every tenant sets this explicitly). A caller can override threshold/timeout knobs per-request, but can't enable a disabled metric or switch tenant by passing input. Verdicts from all requested metrics roll up into one EvaluationResult — an aggregated action (allow/warn/block) and a placeholder 0-100 trust score, not yet calibrated against real incident data.

Project layout

contracts/    shared types + zod schemas — the only thing every folder depends on
metrics/      the escalation-ladder logic per metric (injection, groundedness)
judges/       provider-agnostic judge interfaces + offline stub judges (no API key needed)
policy/       per-tenant policy resolution (in-memory registry for now)
server/       transport-agnostic handlers (flagInjection, checkGroundedness) + trust-score rollup
server/index.ts   stdio MCP entrypoint (Claude Desktop, Cursor, local dev)
app/api/mcp/  HTTP MCP entrypoint (Streamable HTTP, bearer-token auth) — same handlers, thin glue
app/          Next.js demo dashboard
scripts/      smoke test proving the HTTP deploy target end-to-end

The handlers in server/handlers/ are plain async functions with no transport imports (no MCP SDK types, no Next.js types) — both the stdio entrypoint and the HTTP route call the exact same functions. Only the registration glue differs between deploy targets.

Judges are provider-agnostic by interface (judges/types.ts); the only implementation checked in is an offline stub (judges/stub.ts, zero network calls, simple pattern/overlap scoring) so git clone && npm install && npm test works with no API key. A real model-backed judge would implement the same interface in its own file.

Running it

npm install

# stdio MCP server (Claude Desktop, Cursor, etc.)
npm run mcp:stdio

# Next.js app: HTTP MCP endpoint (/api/mcp) + demo dashboard (/)
npm run dev

The HTTP endpoint requires a bearer token — it's a hardcoded token→tenant map for local dev (server/http/auth.ts), not a real authorization server:

curl http://localhost:3000/api/mcp \
  -H "Authorization: Bearer sentinel-local-dev-token" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}'

Open http://localhost:3000 for the demo dashboard: click "Run demo session" to watch a real support-agent transcript scored turn by turn — including a fabricated contract citation and a prompt-injection attempt, both caught before they'd have reached the user. The pass/fail states come from genuine round trips to the real handlers, not mocked UI state.

npm test          # unit + regression tests (metrics, handlers, scoring)
npm run typecheck # tsc --noEmit --strict
npm run smoke:http # end-to-end proof the HTTP deploy target works: real MCP
                    # client, real Streamable HTTP JSON-RPC, auth included

What's a placeholder vs. what's real

• The escalation ladder, policy resolution, and both metrics' detection logic are real and tested.

• The trust score and the weighted aggregation mode are explicit placeholders — a starting point, not calibrated against incident data.

• The policy registry is in-memory; a Postgres-backed implementation implements the same PolicyRegistry interface and should be a drop-in.

• Two more metrics (toxicity, pii_leakage) are already wired into the policy schema as enabled: false — reference points for where their implementations would live, not stubs pretending to work.

• The only judge implementation is the offline stub. A real model-backed judge slots into the same InjectionJudge/GroundednessJudge interfaces.