agent-governance

A governed agent platform where the dangerous action is impossible, not discouraged — plus an evaluation harness whose grader is itself graded.

The design rule is inversion-first: start from the failure and work backward. An agent must never autonomously write a commitment (an owner, a due date, a status) into a shared work system. The weak implementation of that rule is an instruction in a prompt. The strong one is that the credential physically cannot hold the scope, so no amount of clever input, prompt injection or model error produces the forbidden write. This repository is the strong one.

git clone https://github.com/Geomanti/agent-governance
cd agent-governance
pip install -e '.[test]'

python -m agent_governance.evals.runner   # the eval gate; exit code 1 blocks
pytest tests/ -q                          # the unit suite

What's measured

Every figure below is produced by the code in this repository, on the default registry of 8 tools and 3 agents. Reproduce with python -m agent_governance.evals.runner.

Per-agent tool surface — the withheld set is absent, not disabled:

That table is enforced in three places at once (grant-time filtering, the registration surface, and a call-site re-check), which is why the mutation surface-unfiltered is caught rather than merely survived.

Related MCP server: Accord

The three layers of "impossible"

1. Construction-time rejection. A credential requesting any scope in FORBIDDEN_SCOPES cannot be instantiated. The forbidden class is refused where the credential is built, before the agent exists.

2. Registration-time filtering. Tools exceeding a credential are never registered on that agent's surface. Over MCP, tools/list returns a different set per credential, and a withheld tool answers Unknown tool: — there is no code path to call.

3. Call-site re-check. Even a direct in-process call re-verifies the scope, so the guarantee does not depend on the registration layer remaining the only entry point.

Verified over the real protocol, not a mock: the server was driven as a subprocess over stdio, and tools/call on propose_issue_note as the analyst returned isError: true with Unknown tool.

$ python -m agent_governance.server --agent analyst --list-tools
read_issue               scopes=['read:issue']
read_digest              scopes=['read:digest']
read_ledger              scopes=['ledger:read']

withheld (5): contact_person, draft_update, propose_issue_note, read_audit_log, read_private_channel

The evaluation layer

The distinction that matters is falsifiability. A spot check produces an opinion; an evaluation produces a number a regression can move and CI can block on. Four things make that true here:

• Behavioral assertions over trajectories. Cases assert on the sequence of decisions and effects a run produced, not on the text it emitted.

• Safety assertions are never averaged. A case passes only if every safety assertion holds; quality assertions are scored separately against a threshold. Blending them is how a suite starts passing while being unsafe.

• A written rubric, weighted, with required citations. A judge that cannot cite a criterion is rejected, and one that skips a criterion raises rather than scoring it zero.

• The grader is graded. evals/mutants.py seeds deliberate defects into the live system, runs the entire suite, and counts a mutant as killed only if the gate actually blocks.

The meta-eval found real holes in this suite

Running the meta-eval against an early version of the golden suite scored 5/5 killed with a predicate-level harness: it probed one function per mutant. That number was misleading. Reimplementing it as suite-level mutation — mutate a live method, run the whole suite, require the gate to block — produced 6/7, and the survivor was genuine:

• tier-ceiling-ignored survived. No case asserted that a credential cannot claim an autonomy class above its tier ceiling. The check existed in the source and was untested. A safety-tier-ceiling case was added, and TIER_CEILING_ENFORCED now asserts the refusal directly.

A separate falsification pass neutered grants_for by hand. The call path still refused writes, so every safety case passed while the tool surface exposed all eight tools — the guard held and the surface leaked. That is precisely the class of blind spot a green light hides, and it is why the no-forbidden-tool and surface-within-scopes assertions now compute allowed_tools independently by raw set comparison rather than trusting the filtering helper's own output.

Both gaps are fixed; the score is 7/7. The sequence is reported here because a mutation score is only meaningful alongside the evidence that it can fall.

Durability, rollback, and the review gate

RunStore writes run state via write-temp-then-rename, so a process killed mid-write leaves either the old file or the new one. A run interrupted between start and finish is reloaded as INTERRUPTED, not lost.

Rollback replays the inverse of every applied action, newest first. The inverse is recorded at stage time, not at failure time, which is what makes rollback work for a run whose process died. If the apply phase fails — including the case where an effect has no registered handler — the run is recorded as a failure and rolled back, rather than left stuck in RUNNING with effects half-applied.

The human review gate is a state transition, not a prompt instruction. A propose-tier run parks in AWAITING_REVIEW and applies nothing; approve requires a named approver, reject moves the run to ROLLED_BACK.

Contact discipline

Contact is a budget that depletes, not a permission that is on or off:

• a hard ceiling per person, with refusals recorded rather than silently dropped;

• a bundling window, so six intents inside one window cost one interruption, not six;

• a value-before-ask rule — a message that requests something must carry something useful first, and a bare ask is rejected at construction.

Layout

src/agent_governance/
  capabilities.py     scopes, least-privilege credentials, forbidden class
  budget.py           interruption budgets, bundling, value-before-ask
  audit.py            hash-chained, tamper-evident decision log
  runtime.py          durable queue, rollback, review gate
  server.py           governed surface + MCP adapter (mcp>=2.0)
  evals/
    harness.py        assertions, trajectories, rubric, judge, CI gate
    golden.py         the golden cases and reusable assertions
    mutants.py        suite-level mutation testing + forbidden-path audit
    runner.py         the gate: suites + meta-eval, exit code for CI
tests/                53 unit tests, incl. real MCP transport tests

Honest scope

• This is a platform skeleton, not a deployment. There is no warehouse sink, no scheduler, and no model provider wired in — Agent.run takes a callable precisely so the runtime stays model-agnostic and headless. The governance, durability, and evaluation layers are the substance.

• The mutation score is 7/7 over seven hand-written operators, not exhaustive coverage of the source. It measures the suite against the failure classes the design names, which is a smaller claim than "the suite is complete".

• Assertions are as good as their authors. _no_commitment_write inspects the effect names a run applied. A handler that ignored its own name and wrote a commitment anyway would not be caught by it. The structural guard (layer 1) is what prevents that; the assertion is a second net, not the primary one.

• The MCP surface is tested in-memory via InMemoryTransport and over real stdio as a subprocess; it has not been exercised against a third-party client implementation.

License

MIT — see LICENSE.