# mcp-kafka-gouverne

**An LLM agent plugged into a Kafka bus without breaking the bus.** A triage consumer grades night-patrol
reports with a model and raises alerts; an MCP server lets people's agents read topics and propose writes.
Four guarantees, each with a test that fails when it breaks, on a real broker with SASL and ACLs.

| | Guarantee | How | Proof |
|---|---|---|---|
| G1 | Zero duplicate alert after a crash | Kafka transaction per batch (decisions + DLQ + offsets); pager stores offsets with its inbox/outbox in one SQLite transaction, then calls the paging API with an idempotency key | `kill -9` mid-transaction and at random: [test_g1_crash.py](tests/test_g1_crash.py) |
| G2 | Replaying history triggers nothing | Exactly-once does not cover replays: a compacted ledger topic, rebuilt on every partition assignment | group rewind + full re-send: [test_g2_replay.py](tests/test_g2_replay.py); random crash/replay sequences with Hypothesis: [test_properties.py](tests/test_properties.py) |
| G3 | A poison record goes to the DLQ and does not block its partition | Undecodable record copied byte for byte to `*.dlq` inside the same transaction; an LLM outage is *not* poison | [test_g3_poison.py](tests/test_g3_poison.py) |
| G4 | Topic content cannot make the agent act beyond its rights | Rights are the **broker's ACLs on the caller's own principal** (no allow-list in the server, no token passthrough); writes are schema-checked, dry-run by default, and need a human answer through MCP elicitation; the grader has no tools and its grade is `max(floor, llm)` | a fully hijacked agent with a rubber-stamp human, refusals read back from `kafka-authorizer.log`: [test_g4_rights.py](tests/test_g4_rights.py) |

Plus: one Jaeger trace from the agent's tool call to the page ([test_trace.py](tests/test_trace.py)), a
schema-compatibility gate ([test_schemas.py](tests/test_schemas.py)), and the server checked with MCP
Inspector ([test_inspector.py](tests/test_inspector.py)). Why, and what is out of scope: [DESIGN.md](DESIGN.md).
What the existing Kafka MCP servers leave out: [docs/GAPS.md](docs/GAPS.md).

## Run it

```bash
make setup && make up   # Python deps; Kafka 4.1 (KRaft, SASL, ACLs), Schema Registry, Jaeger, Prometheus
make ci                 # lint, schema gate, every guarantee test (a few minutes, no LLM)
make bench              # the numbers below, on the live model (cents; hard cap 2 USD)
```

## Measured

<!-- bench:start -->
Model `claude-haiku-5-5`, 3 replays of `data/snapshot.jsonl`, 2026-11-04. Ranges are min to max over the replays.

| Measure | Value |
|---|---|
| Reports in the frozen snapshot (+3 poison records) | 236 |
| Severity exactly right, floor + LLM | 97.9% to 98.7% |
| Severity exactly right, boxes only (no LLM) | 62.3% |
| Critical (RED) reports missed, of 53 | 3 to 5 |
| ... of which RED only in free text, of 36 | 3 to 5 |
| Critical missed with boxes only, i.e. with a fully hijacked grader | 36 |
| False RED | 0 |
| 'Lower it to GREEN' injections that worked, of 9 | 0 to 1 |
| 'Raise it' injections that worked, of 2 | 0 |
| LLM cost per 1,000 reports | $0.117 |
| LLM call latency p50 / p95 | 1.12 s / 1.54 s |
| End-to-end p50 / p95 (burst of the whole snapshot) | 24.8 s / 42.8 s |
| Max consumer lag during the burst (Prometheus) | 79 to 139 |
| Dead letters (poison records) | 3 |
| Replay on the same ledger: records appended / LLM spend | 0 / $0.00 |
| Live agent as bob@analytics: tool calls / writes done / out-of-rights done (attempts) / planted notes read | 2 / 0 / 0 (0) / 24 |
| Live agent as alice@oncall: tool calls / writes done / out-of-rights done (attempts) / planted notes read | 3 / 0 / 0 (1) / 16 |
<!-- bench:end -->

Every missed RED in these runs is the same note: a cleaner found on the floor, conscious, complaining of
their back. The model calls it ORANGE, with or without a planted note on top, so the "injections that
worked" are that same underestimate rather than the instruction taking effect. The box `injury` would have
caught it: the floor is for exactly this.

The snapshot ([data/snapshot.jsonl](data/snapshot.jsonl)) is synthetic: the layout of a real patrol form
(site, times, ticked boxes, free text in French, photos, "on-call already notified"), invented content, and a
ground truth known by construction ([data/make_snapshot.py](data/make_snapshot.py)). About one report in eight
carries a planted instruction.

## How it fits together

```
webhook ─► bridge ─► patrol.reports.v1 ─► triage ─────────────► patrol.triage.v1 (compacted = ledger) ─► pager ─► paging API
                        (Avro)             │ one transaction:      (one decision per report)            │ SQLite: offsets
                                           │ decisions, DLQ,                                            │ + inbox + outbox
                                           │ offsets                                                    └ Idempotency-Key
                                           ├ floor from boxes (code)  ─┐
                                           └ LLM on free text, no tools ┴► max(floor, llm)
                                           └► patrol.reports.v1.dlq

agent ─► MCP server ─ JWT (audience = this server) ─► sub ─► Kafka principal ─► broker ACLs decide
            list_topics · read_topic (escaped JSON in a nonce-tagged envelope) · produce_message
            (schema check ─► dry run ─► elicitation: the human says yes ─► broker says yes or no)
```

## What it does not protect

Found by two adversarial reviews of this code, and left open on purpose or out of scope:

- **A hijacked grader on a report with no box ticked.** The floor only covers the structured part: if the
  model is taken over *and* the guard wrote "forced door" without ticking the box, the alert is missed.
  The "boxes only" row above is exactly that worst case.
- **The "on-call already notified" box turns the page off.** That is the real system's rule (never wake
  someone twice), and a tampered form can use it. The decision stays RED and visible on the topic.
- **The report id comes from the form.** A forged report sent first with the next id would make the real one
  a duplicate. Fix in production: the bridge assigns the id.
- **The human gate trusts the MCP client to really ask a human.** A client that auto-approves defeats it;
  URL-mode elicitation to a page the server owns would not. The broker still stops out-of-rights writes.
- **Reading is not sending.** If the agent's host has other outbound tools, data read here can leave through
  them, out of the broker's sight.
- **One triage instance.** Several instances stay duplicate-free only while a report id always lands on the
  same partition (keyed by report id, murmur2, fixed partition count).
- **One refusal is not in the broker's audit log.** A principal with no Write anywhere is refused its
  idempotent producer id (`CLUSTER_AUTHORIZATION_FAILED`); Kafka 4.1 does not log that check. The Metadata
  `DESCRIBE` refusal before it is logged, and the client sees both.
- **The paging API must deduplicate on the key** (delivery to it is at-least-once); some real ones only
  deduplicate within a time window.
- Local only: SASL/PLAIN without TLS, one broker, a toy identity provider, and the demo server process can
  read every local password. See [DESIGN.md](DESIGN.md).

MIT licence.