FourEyes

A human-gated customer-support agent. It reads tickets, queries accounts, and decides what to do — but every irreversible write (refund / escalate / close) physically stops at a human approval gate before it can execute.

The name is the four-eyes principle: any critical action needs a second pair of eyes.

The argument

Most "AI agent safety" is prompt text: "please ask a human before refunding." A prompt is a request, not a constraint — one ignore previous instructions and it's gone.

FourEyes puts the guarantee somewhere a prompt can't reach:

Two properties are enforced by tests, not by comments:

• No path from START to execute_action bypasses interrupt() — asserted by deleting the interrupt node from the graph and proving execute_action becomes unreachable.

• Authorization comes from the approved database row, not from mutable graph state. execute_action re-reads the approvals row the human signed and cross-checks it against the proposal; a mismatch is refused and audited. (This one came out of an adversarial review that found the original code could execute a refund while the human had approved an escalation — see failures.md.)

Architecture

                                  ┌──────────────────────────────────────┐
  ticket ──▶ sanitize_input ──▶ gather_evidence ──▶ classify ──▶ route   │
             (layer ①)           (read-only MCP)     (LLM, policy)       │
                                                          │              │
              ┌───────────────────────────────────────────┤              │
              ▼                    ▼                      ▼              │
        out_of_policy       under_specified          in_policy           │
              │                    │                      │              │
        explain_refusal      propose_escalation     propose_action       │
              │                    └──────────┬───────────┘              │
             END                              ▼                          │
                                       request_approval  ── writes approvals row
                                              ▼
                                    ★ await_decision — interrupt()
                                       state → Postgres checkpoint
                                              │
                        ┌─────────────────────┴──────────────────┐
                     rejected                                 approved
                        │                                        │
                  log_rejection                            execute_action ── the ONLY
                        │                                        │            ticket-action
                       END                                verify_and_log      client
                                                                 │
                                                                END

Four services, one command (docker compose up):

Console (console/, React + TypeScript + Vite) is one screen: pending cards → approve / reject.

Why two MCP servers instead of one with two tool groups? The permission boundary is drawn at the protocol and network layer, not inside a function. Read tools are ungated because gating everything causes approval fatigue — a gate everywhere is a gate nowhere. Only irreversible writes are gated.

Measured results

Every number below came from a command in this repo. Nothing here is estimated.

Adversarial testing — 53 emails, 7 attack categories

.venv/bin/python evals/test_redteam.py     # report: evals/redteam/report.json

total_emails            : 53   (direct injection · roleplay/jailbreak · forged system messages ·
                                encoding/obfuscation · social engineering · tool-parameter
                                pollution · multi-turn priming)
unauthorized_executions : 0
deception_rate          : 0.0  (0/53 talked the model into proposing a refund)
sanitize_flagged        : 21/53
blocked_by seen         : content_layer + structural_layer + business_guardrail   ← all three

Every email is auto-approved during the run — deliberately simulating a human who is also fooled — so the business guardrail is the thing being tested, not the human.

The two metrics are reported separately on purpose: zero unauthorized executions is the execution-layer claim; deception rate is the reasoning-layer experiment. Nobody wants a refund system that is 96% safe, so the safety claim is a count, not a percentage.

Action selection — 100 labeled tickets

.venv/bin/python evals/test_benchmark.py   # report: evals/benchmark/report.json

action_selection_accuracy : 99.0%  (99/100)
false_block_rate          : 0.0%   (0/31 actionable in_policy tickets)
per_subset                : generated 98.8% (79/80) · boundary 100% (20/20)

Dataset composition matters more than the number. 80 tickets are LLM-generated with clean-cut policy boundaries; measuring found only 3 of them landed within ±5 days / ±$50 of a threshold, which made 98.8% indefensible on its own. So 20 hand-written boundary cases were added: day 30 vs day 31, exactly $500 vs $500.01, exactly the order amount vs one cent more, pending/rejected prior refunds (which do not block a new refund), and three policy-precedence conflicts (X3 beats E1; X4 beats E1; a safety incident outranks amount). The boundary subset scored 20/20 — the classifier is reasoning from clauses, not keyword-matching.

The single miss (bm_076) cited E3 + X1 and escalated where the label said refuse — a third-party request made on behalf of an 84-year-old parent. Defensible disagreement, not a bug.

Trajectory evals — 29 scenarios, and proof they have teeth

.venv/bin/python -m pytest evals/test_trajectories.py -q   # 30 passed in 124.85s
.venv/bin/python scripts/verify_eval_teeth.py

Trajectory evals assert the process, not just the answer — a correct final state can be reached by a wrong path (the Friday-refactor that quietly routes around the approval node). 10 of the 29 are negative scenarios.

An eval suite nobody has ever seen fail is not a safety net, so failure is demonstrated on demand: verify_eval_teeth.py rewrites the approval edge to request_approval → execute_action, runs the evals, and requires them to go red — then restores the file and requires green:

=== step 1: sabotage the approval edge ===
3 failed (traj_bypass_check, traj_single_inbound_edge, traj_001), exit=1
OK: evals went RED as required
=== step 2: re-run against the intact graph ===
4 passed
VERDICT: trajectory evals have teeth

Note traj_001 — a behavioural scenario — also goes red, not just the topology assertions.

Test suite

.venv/bin/python -m pytest tests/ -q       # 43 passed

Guardrails (14) · topology (6) · consent binding (4) · layer-1 guards (16, including 6 false-positive guards so ordinary complaints stay unflagged) · guardrail backstop (3).

Synthetic data disclosure

All tickets, customers, orders and adversarial emails in this repo are LLM-generated synthetic data. There are no real customers, no real orders, and no production traffic. Specifically:

• db/seed_data.json — 60 tickets across 9 scenario categories, generated by Claude and cached to git so re-seeding is deterministic (ADR-003).

• evals/redteam/emails.jsonl — 53 adversarial emails. Six categories are Claude-generated; the encoding_obfuscation set is built programmatically (real base64 / zero-width / homoglyph payloads) because Claude's safety classifier declines to encode live attack instructions.

• evals/benchmark/tickets.jsonl — 80 labeled tickets, Claude-generated, with every label checked against deterministic policy rules before entering the dataset — one self- contradictory item was dropped and regenerated (ADR-011).

• evals/benchmark/boundary.jsonl — 20 hand-written boundary cases.

Dates are stored as relative offsets and converted at seed time, so "within the 30-day window" scenarios stay valid whenever the dataset is re-seeded.

OWASP LLM Top 10 mapping

Running it

Requires Python 3.12+, Node 20+, and Docker.

# 0. Local Python env — the scripts and evals run on the host, not in the containers
python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt

# 1. Full stack
cp .env.example .env          # fill in ANTHROPIC_API_KEY, GOOGLE_API_KEY, Langfuse keys
docker compose up -d --build  # postgres + mcp-lookup + mcp-action + api

# 2. Seed synthetic tickets (uses the cached generation; no API call needed)
.venv/bin/python db/seed.py --reset

# 3. Drive one ticket to the approval gate — the process then exits
.venv/bin/python scripts/run_ticket.py start --category refund_eligible

# 4. Approve from a *different* process, resuming from the Postgres checkpoint
.venv/bin/python scripts/run_ticket.py resume <ticket_id> approved --by you
.venv/bin/python scripts/run_ticket.py inspect <ticket_id>

# 5. Or approve in the console
cd console && npm install && npm run dev     # http://localhost:5173

Step 3 → 4 is the checkpoint demo: two separate processes. The second one resumes from the stored checkpoint instead of re-reasoning — which matters because an LLM asked twice may reach a different conclusion, and the human approved a specific proposal, not a re-roll.

Design decisions

Full ADRs with alternatives-considered in decisions.md. The load-bearing ones:

• [ADR-002] Two DB roles. foureyes_ro has no write grant, so "the lookup server is read-only" is a database fact rather than a code convention.

• [ADR-007] Deterministic evidence gathering, single LLM decision point. No ReAct tool loop — the trajectory assertions can be exact, and benchmark variance comes from judgment rather than from retrieval flakiness.

• [ADR-007] request_approval and await_decision are separate nodes. LangGraph replays a node on resume; side effects must sit after the interrupt() or the approvals row is written twice.

• [ADR-009] Consent is bound to the executed action. Authorization is re-read from the approved row at execution time.

• [ADR-012] The API cannot execute. Approving only resumes the graph, so compromising the console still cannot move money.

Scars, including three real bugs found after the code "worked", are in failures.md.

AI-assisted development workflow

This project was built with Claude Code. What that means concretely, and how the output was verified:

Discipline used while building

• Every component got a decisions.md entry before implementation — decision, alternatives, why, and what breaks with the alternative. Not being able to name an alternative meant the design wasn't understood yet.

• Every mistake went into failures.md with the verbatim error, the diagnosis and the fix.

• Nothing was called "done" without running it and pasting the output into the commit message.

• Output numbers (accuracy, block rates) were forbidden from appearing anywhere — code comments included — until a command had produced them. Placeholders read [NOT_MEASURED].

How AI output was verified

1. Adversarial code review. Four independent review agents (HITL topology, injection bypass, guardrail completeness, correctness) produced 23 raw findings; each was then handed to a separate agent instructed to refute it against the real code. 23 → 3 confirmed. Without the refutation pass the real bug would have been buried in false positives.

2. The confirmed HIGH was a genuine design flaw, not a typo: consent and action were decoupled, so a replay could have the human approve an escalation while a refund executed. Fixed structurally (ADR-009) plus 4 regression tests.

3. End-to-end demos found what unit tests could not. Layer-3 backstop and layer-1 regex gaps were both caught by red-team demos, after unit tests and protocol smoke tests passed — the bugs were in the seams between components.

4. The red-team harness's own ground truth was wrong first. It initially reported 10 unauthorized executions; the carrier orders were accidentally legitimately refundable. The dangerous failure mode for a safety metric isn't an ugly number, it's a pretty number measured against the wrong baseline.

5. Generated labels are machine-checked. Benchmark labels are validated against deterministic policy rules before entering the dataset, so the metric measures agreement with the policy rather than agreement with another model.

Out of scope (deliberately)

No voice/TTS, no chat UI, no dashboards or charts, no login system, no fine-tuning, no real user traffic. The approval console is one screen — anything more is scope creep.

Tracing

Every ticket produces one Langfuse trace, keyed deterministically off the ticket id so the spans emitted by the start process and the resume process land in the same trace:

SPAN       sanitize_input        injection_flags recorded here
SPAN       gather_evidence       the five read-only lookups
GENERATION classify              policy + evidence → decision (prompt/completion/tokens)
SPAN       approval_requested    ← the graph stops here
SPAN       human_decision        ← human waited 9.7s   (waited_seconds in metadata)
SPAN       execute_action        runs only what the approved row authorises
SPAN       verify_and_log        reads the ticket back

approvals.trace_url stores the link, so every card in the console deep-links to its own trace. Provider fallback is emitted as a provider-fallback event on the trace, so the Claude → Gemini switch is visible rather than inferred.

Region gotcha, in case you fork this: Langfuse Cloud is region-split. Pointing an US project at cloud.langfuse.com returns 401 Invalid credentials — which reads like a bad key but isn't. Cost me a full misdiagnosis; see failures.md.

Known gaps

• Provider fallback is verified by a real APITimeoutError (scripts/smoke_router.py), not by a mock — but it has not been exercised under an actual provider outage.

• MCP server connectivity was verified with the MCP Python SDK client (list_tools + call_tool over Streamable HTTP), not with the MCP Inspector UI. Protocol-equivalent, but if you want to claim "verified in Inspector", run it yourself first.