bizops-mcp

An evaluated, multi-tenant control plane for AI agents acting across business systems.

The question this repository answers with measurements: how can an AI agent investigate and act across a company's systems without huge tool schemas in every request and without unauthorized or duplicated side effects? The agent gets three MCP tools (discover, execute, inspect). Behind them sit server-side policy, single-use human approvals, a durable journal with server-owned operation ids, failure injection, OpenTelemetry tracing, replay, and BizOpsBench: 100 scenarios over a deterministic synthetic SaaS company, graded from the database, run on Claude Sonnet 5.5 against two baselines at 27 and 500 capabilities. Headline: at 500 capabilities the discover surface matches one-tool-per-capability on success (98% vs 99%, paired difference −1 point, 95% CI −6 to +3) at 1/14th the cost ($0.0154 vs $0.2119 per task), and every control-plane layer removes a failure the ablation can count.

$ bizops serve --transport http            # or stdio for Claude Code / Claude Desktop
agent → discover("refund a payment")
        billing.refund_payment(payment_id: str, amount_cents: int, reason: str) -> Refund
          [financial; approval over $5,000; scope billing.read, billing.refund]
agent → execute(steps=[{capability: billing.refund_payment, args: {payment_id: py_0228, amount_cents: 2120800, …}}])
        ⏸  human asked: "Approve billing.refund_payment({…})? Why: 21,208.00 exceeds the 5,000 threshold"
        ✓  approved → run_3f2a… done, refund re_0001, exactly once even if the response is lost

The synthetic company is a fixture. The control plane, the execution semantics and the benchmark are the project.

What is in the box

Related MCP server: foral

Architecture

src/bizops/
  domain/          models (Capability, Principal, Run, Step), policy, roles, signatures, ports   no I/O
  application/     catalogue, discovery, executor + dispatcher, approvals, surfaces, replay, telemetry
  infrastructure/  SQLite tenants + journal, connectors (billing, crm, support, usage, engineering, people),
                   seed fixture, chaos, settings, container, tracer setup
  interfaces/      MCP server (stdio + HTTP), CLI (seed, serve, approve, replay)
bench/             scenarios, checkers, harness (agent loop, grader, stats, report), scaling experiment

The dependency rule points inward and is a test, not a convention: tests/unit/test_architecture.py fails CI if the domain imports anything outward or the application imports infrastructure. A package exists only when it holds several modules. Details: docs/ARCHITECTURE.md. Threats and controls: docs/THREAT_MODEL.md.

Quick start

uv sync --all-extras
uv run bizops seed --tenant acme
uv run bizops serve --principal acme/carol                 # MCP over stdio
uv run bizops token issue acme/carol                       # prints a random token once; only its hash is stored
uv run bizops serve --transport http --port 8765           # Authorization: Bearer <token>
make check                                                 # ruff, mypy strict, 279 tests, no API key
uv run python -m bench run --scenarios bench/scenarios --arms static,tool_search,discover --catalogue 500

Evaluation

Claude Sonnet 5.5 (claude-sonnet-5-5, effort medium), one manual tool loop for every arm, identical tasks, seed, principal, policy, connectors and chaos; only the agent-facing surface changes. 690 runs, an estimated $33.07 at list prices as of 2026-10-06 (costs below are estimates from token counts), no run excluded for API failures. Intervals are 95% paired bootstraps over 52 template families, not 100 scenarios, because parametrised cases of one template are not independent. Methodology: docs/BENCH.md. Tables are pasted from bench/results/full/REPORT.md, never typed by hand.

Table A: the interface (full control plane behind every surface)

27 capabilities (the real catalogue)

500 capabilities (27 real plus 473 near-miss distractors such as accounting.invoices.list)

Discover retrieval over the 92 scenario tasks, each searched as one query (bench/results/scaling/scaling.json): recall@8 98.9% at 27 capabilities, 93.5% from 250 up to 973; recall@3 falls to 82.6% at 500.

Table B: what each control-plane layer prevents (discover surface)

The ablation rows run the 45 write, approval, failure and security scenarios with layers switched off; the full row is every discover run at both sizes (200), where those errors could also have appeared.

Invariant gates on the full plane, all 600 Table A runs: cross-tenant leaks 0, forbidden capabilities executed 0. The full plane's 2 "uncertain writes" are the lost-response ticket replies ending in delivery_uncertain by design.

Read it honestly

• Static did not get less accurate with 500 tools. Sonnet 5.5 solved 99% of the tasks with 500 schemas in context (trajectory correct 96%, the same as at 27). The thesis that a large static surface hurts success is not supported here; what it hurts is cost: 215k billed input tokens and $0.21 per task, 14 times discover.

• Discover's win is cost at equal success, about half the tokens at 27 capabilities and a fourteenth at 500, with more calls (3.6 vs 3.0) and a slower median than static at 27.

• Anthropic tool search has the smallest surface (643 tokens) and the lowest success. Its typical failure: the search loads the target tool (billing.subscription.get, crm.contacts.list) but not the name-to-id lookup it needs first, and the model, correctly, refuses to guess an account id. Discover returns eight signatures per query, so the lookup usually comes along. At 500 capabilities the difference against discover (+6 points) is not significant.

• The benchmark found a real bug. In the first full run, two $50 refunds of one payment with different free-text reasons ("1 of 2", "2 of 2") both went through without a human: the duplicate guard compared all arguments. Capabilities now declare the arguments that define their effect; the scenario was re-run and passes on every surface, and the original rows are kept in superseded.jsonl (#16).

• Approvals are the cheapest layer with the biggest effect; the journal is what stops duplicates under retries and lost responses, and nothing else in the stack does.

• The pilot's two repeats showed no run-to-run variance on these tasks, so the full run uses one repeat.

Limitations

One model. Synthetic data and a simulated human who always answers as the scenario says. Answers are graded by string matching of checker facts (dates and cents normalised), which is strict and occasionally harsh on paraphrase. Tenant isolation is physical (one SQLite file per tenant), so the cross-tenant gate is zero by construction; the scenarios check that nothing leaks through the answer. Discover is lexical BM25; the scaling experiment shows where recall starts to fall. No OAuth server and no MCP Tasks: authentication is simplified on purpose, and durable execution is bizops-native.

Status

• fixture, 27 capabilities over six connectors, server-side policy

• durable executor, single-use approvals, failure injection, crash-after-commit adoption

• discover / execute / inspect and static surfaces over stdio and Streamable HTTP with bearer principals

• OpenTelemetry tracing and bizops replay

• BizOpsBench: 100 scenarios, two catalogue sizes, ablation, schema-scaling experiment

• an Opus row, and Sonnet + discover against Opus + static

• embedding retrieval behind discover if the large-catalogue recall drop matters in practice

Development

make check
uv run python -m bench.harness.generate     # regenerate the 85 parametrised scenarios
uv run python -m bench report bench/results/full

MIT.