boardroom-agents

A small Python multi-agent system that answers questions about a board's history, where a dedicated governance agent evaluates every drafted answer against an externalized policy before the answer is returned — allow, revise, or block.

It reads the SQLite database from boardwalk (a board-management web app) read-only. The interesting property is inherited from that schema: individual anonymous votes cannot be revealed, because they were never recorded — anonymity is enforced by the data model, not by a prompt. The governance agent is a second, independent line of defense on top of that structural guarantee.

I took a governance app whose guardrails are enforced by the schema, not the prompt, and refactored its AI into a multi-agent system where a dedicated governance agent judges every answer against an externalized policy before release — allow, revise, or block. That is conceptually what agent-governance products do: evaluate an agent's action against policy before it executes, with an audit trail.

Runs fully offline with no API key (PROVIDER=mock is the default).

Architecture

                      question
                          │
                ┌─────────▼──────────┐
                │    Orchestrator    │  bounds the run (global step budget),
                │  orchestrator.py   │  owns retry / escalation / final answer
                └─────────┬──────────┘
                          │ (1) delegate
                ┌─────────▼──────────┐        ┌──────────────────────────────┐
                │   Research agent   │◄──────►│  Audited tool layer          │
                │    research.py     │ bounded│  tools.py (5 tools)          │
                │                    │  loop  │   list_meetings              │
                │  drafts an answer  │ MAX_   │   get_meeting                │
                │  WITH citations    │ TOOL_  │   list_motions               │
                └─────────┬──────────┘ TURNS  │   get_motion_results  ◄── the│
                          │                   │   search_documents   anonymity
                          │                   └───────────┬──────────┘ branch │
                          │                               │ read-only         │
                          │                   ┌───────────▼──────────┐        │
                          │                   │  data/board.db       │        │
                          │                   │  sqlite mode=ro      │        │
                          │                   └──────────────────────┘        │
                          │ (2) submit draft + the evidence actually used
                ┌─────────▼──────────┐        ┌──────────────────────────────┐
                │  Governance agent  │◄───────│ policies/answer_policy.md    │
                │   governance.py    │  reads │ (externalized, hot-editable) │
                │                    │ at run-└──────────────────────────────┘
                │  LLM-as-judge  +   │  time
                │  deterministic     │
                │  checks (the floor)│
                └─────────┬──────────┘
                          │
              verdict ────┼──────────────────────────────────
                          │                │                │
                      allow                revise           block
                          │                │                │
                  return answer   feed judge feedback   return the policy's
                                  back to Research      safe refusal — the
                                  (MAX_REVISIONS)       draft is never shown

Every step is appended to a Trace, which the CLI prints before the answer. That visible reasoning — routing, tool calls, draft, verdict — is the demo.

Why three agents

Two entry points over one governed core

The loop above is reached two ways. cli.py drives an agent whose code is in this repo. The MCP server serves agents that aren't — and that difference is the whole point of having it.

      human at a terminal                          any MCP client
      (agent loop you wrote)              (Claude Desktop / Code / internal)
               │                                          │
         cli.py│                                          │ stdio · JSON-RPC
               │                                 ┌────────▼─────────┐
               │                                 │  mcp_server.py   │ ← audit log
               │                                 │   THE BOUNDARY   │ ← egress screen
               │                                 └───┬──────────┬───┘
               │                        ask_board     │          │  5 audited tools
               │             ┌────────────────────────┘          │  (raw records)
               │             │                                   │
      ┌────────▼─────────────▼───────────────────────────┐       │
      │  Orchestrator → Research → Governance            │       │
      │  the governed core — one implementation, shared  │       │
      └────────────────────────┬─────────────────────────┘       │
                               │                                 │
                               └────────► tools.py ◄─────────────┘
                                              │  read-only
                                     ┌────────▼─────────┐
                                     │  data/board.db   │
                                     └──────────────────┘

Note the asymmetry: ask_board routes through the full governance loop, while a raw tool call gets the egress screen and the audit entry but no judge — because there is no answer to judge, only records whose safety the schema already settled.

Related MCP server: ToolBridge

Quickstart

uv sync
uv run python cli.py "What was the result of motion #1?"

No API key needed — PROVIDER=mock uses a deterministic offline client that exercises the real tool layer, the real agent loop, and the real governance checks.

To use a live model instead, copy .env.example to .env and set PROVIDER=openai plus OPENAI_API_KEY (uv sync already installed the SDK). The deterministic governance checks still run underneath the LLM judge; a model can only tighten a verdict, never loosen it.

The judge is told each motion's ballot mode as an established fact rather than being left to infer it from the evidence JSON. Without that it reads "a person is named next to a vote" as an anonymity breach and blocks correct answers about recorded votes — where attribution is the entire point. Facts the database has already settled should be stated to a reviewer, not deduced by one.

The four demo questions

All four work offline.

Question 2 and question 4 are the pair worth showing together: the same tool, the same agent, and the difference in what comes back is decided by the ballot mode recorded in the schema — not by how the question was phrased or how well the prompt was written.

How governance decides

policies/answer_policy.md holds five rules with stable ids. The abstention phrase and the safe refusal are parsed out of that markdown, not duplicated in code — edit the file and the next run behaves differently, with no code change. The rules map to checks like this:

Two design choices are worth calling out, because they're the ones an interviewer should push on:

Attribution is default-deny. The check does not ask "does the evidence prove this ballot was anonymous?" — it asks "does the evidence prove it was recorded?" The inverted version looks equivalent and isn't: a model that attributes a vote while citing a motion it never fetched would pass it. Absence of proof is not permission. Blocking on ambiguity costs a revision round; the other error is unrecoverable.

The deterministic checks are the floor, not the fallback. They run under every provider, including the offline mock, with no network. When a live model is configured it judges too — but it can only tighten a verdict, never loosen one. A control that works only while the model cooperates isn't a control; tests/test_governance.py pins this by handing the judge a leaky draft and having it vote allow, then asserting the answer is still blocked.

Governing an agent you didn't write: the MCP server

uv run python mcp_server.py       # stdio MCP server

cli.py governs an agent whose loop is in this repo. The MCP server governs one that isn't. Point Claude Desktop at it and Claude becomes the connected agent — ask it "who voted against David Lee?" and the refusal comes from the boundary, not from Claude's own judgment. The guardrail travels with the data instead of living in someone else's prompt.

{
  "mcpServers": {
    "boardroom": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/boardroom-agents", "run", "python", "mcp_server.py"]
    }
  }
}

What happens at the boundary

  MCP client                        mcp_server.py
  ──────────                        ─────────────
  initialize ──────────────────►  server info, capabilities
  tools/list ──────────────────►  5 audited tools + ask_board
                                  (descriptions single-sourced from TOOL_SPECS,
                                   so this surface and the OpenAI function-calling
                                   path can never describe a tool differently)
  resources/list ──────────────►  policy://answer-policy
  resources/read  ─────────────►  the full policy text
                                  (inspect the rules you are being held to)

  tools/call ──────────────────►  ┌─────────────────────────────────────┐
   get_motion_results(3)          │ 1  tools.dispatch()  →  raw result  │
                                  │ 2  screen_egress(result)            │
                                  │ 3  audit.jsonl  ← decision + shape  │
                                  └──────┬───────────────────────┬──────┘
                                         │ clean                 │ violation
                                         ▼                       ▼
                                  totals + turnout        BLOCKED_RESPONSE
                                  (no voters)             {error, violations}
                                                          — no records at all

  tools/call ──────────────────►  ask_board("who voted against David Lee?")
   the governed path                     │
                                         ▼
                              Orchestrator → Research → Governance
                                         │
                                         ▼
                              { answer, verdict, violations, reasons, trace }
                              block → the policy's safe refusal, never the draft

It exposes the five audited tools plus ask_board (which routes through the full research + governance loop and returns the verdict alongside the answer), and the policy itself as a readable resource — a client should be able to inspect the rules it's being held to.

Two things happen here that don't happen inside the agent loop:

• An append-only audit log (logs/audit.jsonl). The in-process Trace is per-run and vanishes; this is the durable record of what a connected agent actually did. It stores what was asked and what was decided, plus the shape of each response — never the response. An audit trail that copies the records it protects becomes a second place to leak them.

• An egress screen on every result. The schema already makes an anonymous vote unattributable, so this filter should never fire — which is precisely why it's worth having. It's the assertion that catches a future query, tool, or migration that weakens the guarantee, at the last point before data leaves the process.

Be precise about the claim: exposing these tools over MCP adds no new data safety, because the schema already provides it. What's new is the enforcement point, the audit trail, and request-level governance for a client whose prompt you don't control.

Tests and evals

uv run pytest -q                          # unit tests: tools, anonymity, governance, MCP, protocol
uv run python evals/run_evals.py          # golden set — the gate, pinned offline
uv run python evals/run_evals.py --live    # same questions against the configured provider
uv run python evals/redteam.py            # adversarial probe; non-zero exit if anything leaked

The gate is pinned offline. tests/conftest.py pins every test to the mock provider, and run_evals.py does the same by default. A regression gate whose verdict changes because a model reworded a sentence isn't a gate; it's a coin flip that occasionally fails your build. It also means the suite makes no billable API calls — which it silently did until I noticed the runtime.

--live runs the same questions against the real provider but checks only what a model cannot legitimately vary: the verdict, the rules that fired, whether the answer is the refusal or the abstention, and — always — that no director's name leaked. Wording and citation assertions are skipped there, because "the model phrased it differently" is not a governance failure. What must never differ is the decision.

The red-team probe and the skill that drives it

evals/redteam.py runs 20 attacks on the anonymity invariant across five tactics — direct, indirect inference, authority and pretext, framing and roleplay, and aggregation — and exits non-zero if any answer paired a director's name with a vote on an anonymous ballot.

The skill exists because the parts a script can't do are the parts that matter: inventing attacks that aren't already in the file, and deciding whether a reported leak is real.

  .claude/skills/redteam-anonymity/SKILL.md
   the judgement: when to run, what to write, how to triage, where to fix
        │
        │  1. run the corpus            2. write NEW attacks         3. triage
        ▼                                 (a static list goes stale
  evals/redteam.py                         the moment it is read)
   20 attacks × 5 tactics ──┐           uv run … --file attacks.txt
                            │
                            ▼
                   Orchestrator → Research → Governance
                            │
                            ▼
                     attributed(answer)  +  was an anonymous ballot in play?
                            │
        ┌───────────────────┼────────────────────────┐
        ▼                   ▼                        ▼
    REAL LEAK        recorded ballot            detector noise
        │            → permitted, not a        → fix redteam.py,
        │              finding at all            not the system
        │
        │  4. fix at the deepest layer that can hold it
        │     tools.py  >  governance checks  >  policy  >  prompt
        ▼
   --promote ────► evals/golden.jsonl ────► run_evals.py + pytest
                   a finding cannot be made twice

That last arrow is the point of the whole thing: a red-team finding that isn't written down is a finding you get to make again.

Triage is the part that matters. An early version of the probe flagged three leaks that were all its own false positives — one of them reading "nominations for the office of Board Chair; David Lee" as an attribution, because bare for was in its vote-token list. A probe that cries wolf is worse than no probe, because it teaches you to skim past it. The detector now requires a strict vote token and only counts an attribution as a leak when an anonymous ballot was actually in play — attribution on the recorded motion is correct behaviour and is reported as permitted.

Current state: 20/20 held on the mock, 20/20 on gpt-4o-mini — 7 versus 17 blocked by governance respectively. Report the provider with the score; "20/20 held" means nothing without it, and the two numbers differ because a live model explores more tools and trips more checks.

What running the live gate actually found

Worth telling this story, because it's the most useful thing the evals did. Running the golden set against the real model for the first time gave 4/10, and the failures split three ways:

Fixing the over-blocking then exposed the opposite failure: the money-demo question came back allow. The G2-request check had required an anonymous motion in the evidence, and a live agent doesn't always fetch one. It's now default-deny like the leak check — asking who voted is only safe once the ballot is known to be recorded.

And fixing that over-blocked "How did the budget vote go?", because read case-insensitively how did <two words> vote swallows it. Asking how a vote went is a totals question. The named form is now case-sensitive on the subject, so it needs something that looks like a person.

Three rounds, each fix revealing the next error in the opposite direction. Over-blocking is the quieter failure — nothing leaks, so nothing alarms — and it's the one that makes the system useless by denying the board its own records. Both directions are now pinned in tests/test_governance.py.

tests/test_anonymity.py asserts the structural guarantee: no tool can return a voter paired with an anonymous choice. tests/test_governance.py asserts the judge blocks a deliberately leaky draft, asks for a revision when a factual claim has no citation, and allows a well-cited answer. evals/ is the regression gate — it runs whole questions end-to-end through the orchestrator and checks properties of the final answer.

Why the anonymity invariant is structural, not prompted

In the source schema, recorded and anonymous ballots live in different tables:

• recorded_votes(motion_id, voter_id, choice) — attribution is the point.

• ballot_participation(motion_id, voter_id) — records that a director voted.

• anonymous_ballots(id, motion_id, choice, candidate_id) — the choice, with no voter column at all, a random text primary key, and no timestamp, so there is no insertion-order or time signal to correlate a ballot back to a participation row.

So "who voted against this?" on an anonymous motion is not a question the system declines to answer — it is a question the database cannot answer. get_motion_results' anonymous branch reads only those two no-voter tables. A prompt-injection attempt, a jailbreak, or a bug in the agent loop cannot surface data that was never stored.

The governance agent exists because that structural guarantee, while strong, only covers the data layer. A model can still infer, speculate, or launder an implication ("with 4 of 6 votes and David Lee absent from the abstention…"). The judge is the control that catches reasoning about the data, on top of the schema that controls the data itself.

Layout

cli.py                      entry point: prints the trace, then the answer
mcp_server.py               MCP boundary: audit log, egress screen, ask_board
policies/answer_policy.md   the governance policy, read at runtime
data/board.db               the board database (read-only fixture)
src/boardroom/
  config.py                 env, paths, and every loop budget
  db.py                     read-only SQLite connection
  tools.py                  the five audited tools + JSON specs + dispatcher
  llm.py                    client protocol, deterministic mock, OpenAI client
  trace.py                  structured step log
  agents/
    orchestrator.py         routes, bounds, decides what is returned
    research.py             tool-calling loop, drafts with citations
    governance.py           LLM-as-judge + deterministic checks
tests/                      pytest, incl. real-stdio MCP protocol tests
evals/
  golden.jsonl              known-good questions + expected properties
  run_evals.py              the regression gate
  redteam.py                adversarial probe on the anonymity invariant
.claude/skills/
  redteam-anonymity/        when to probe, how to triage, where to fix
logs/audit.jsonl            written at runtime by the MCP boundary (gitignored)

Three surfaces, one governed core: cli.py for a human, mcp_server.py for an agent you don't control, and evals/ for the machine that tries to break both.

Deliberately not here: no web UI, no auth, no write paths, no vector search, no agent framework. The orchestration loop is written out by hand so it can be read top to bottom.

See PLAN.md for the full specification and CLAUDE.md for the invariants.