# benthic-mcp

A read-only [MCP](https://modelcontextprotocol.io) server over signed [Benthic Data
Provenance](https://benthic.io/bdp/) datasets, plus the evaluation harness that measures whether the
playbook it serves actually improves an agent's answers.

The server exposes six tools:

| tool | what it does |
| --- | --- |
| `benthic_discover` | finds the smallest relevant signed relation, its columns, and its join paths |
| `benthic_playbook` | returns dataset access conventions, or the signed route between two relations |
| `benthic_query` | runs a bounded single-relation query with compact filter, aggregate, and `having` expressions |
| `benthic_join` | runs a multi-relation plan using only signed BDP join paths |
| `benthic_rpc` | runs three allowlisted spatial RPCs |
| `benthic_report` | records one mistake and its correction, held pending until verified and A/B measured |

Plain-language questions are interpreted by the calling chat model, which turns a question into compact
tool arguments. This service validates and executes those arguments without embedding a model of its
own, so what an agent is *allowed* to ask is decided by a signed manifest rather than by a prompt.

## What the server will and will not answer

The interesting part of this server is not what it serves but where it stops, and the refusal is
designed to be terminal rather than a hint the model talks its way past. A question that cannot be
answered from the signed catalog gets "this cannot be answered" and an instruction to say so - not a
suggestion to narrow filters that cannot help.

- **A `max` or `min` is a top-1, not a scan.** One indexed page, `order=col.desc.nullslast limit=1`.
  The complete-scan limit does not apply to it, because it has one output row however wide the source
  is. `.nullslast` matters: Postgres sorts NULLS FIRST on a descending sort, so without it a
  null-heavy column answers `null` instead of the extreme.
- **A bare `count(*)` and a `count(col)` are HEAD counts.** Both have one output row, so neither is
  refused for size. `count(col)` sends `col=not.is.null` on the same HEAD, which is exactly the
  non-null count a scan would compute.
- **An aggregate over a source that cannot be read in full is refused**, and the message names the
  real count that makes it impossible rather than telling the caller to narrow.
- **A relation with no primary key cannot be paged deterministically**, so an aggregate over one is
  refused however far it is narrowed - getting under the scan limit does not help, and no sequence of
  queries reaches a cross-row total there.
- **A complete join result of identical rows says so.** One left row matching every right row is a
  fan-out, and the result carries the representative row plus the exact `row_count`, so a complete
  result cannot read as a truncated page.
- **Discovery payload omits nulls.** Absence and `null` say the same thing, and the payload is
  re-sent to the model on every remaining turn.

Every refusal was written because the alternative - a wrong number that reads as a result - is worse
than a clean stop.

## Why the trust model is the point

The catalog is signed. The server pins an Ed25519 authority key, verifies RFC 8785 canonical JSON,
SHA-256 payload hashes, and every member manifest against the collection hash, and then serves only
what the signature covers:

- only relations marked `queryable: true` in a signed manifest
- only join paths present in the signed join graph
- only an allowlist of RPC endpoints, hand-written in `src/benthic_mcp/catalog.py`
- no arbitrary URLs, relations, SQL, or write operations
- API and RPC traffic restricted to HTTPS endpoints on `benthic.io`

The playbook this server serves to the model is prose, and prose is model-authored. It can therefore
never widen any of the above. A fact the signed catalog already carries is always read from the
verified manifest at serve time, so guidance cannot contradict it. See
[`SECURITY.md`](SECURITY.md) for the trust boundary in one page.

## Requirements

- Python 3.12 or newer
- [`uv`](https://docs.astral.sh/uv/)
- Network access to `https://benthic.io`
- A local [`llama-server`](https://github.com/ggml-org/llama.cpp) for the agent that calls the tools

```sh
git clone https://github.com/benthic-io/benthic-mcp
cd benthic-mcp
uv sync --frozen
```

## Running it

Two transports are supported, and they are not interchangeable with the two MCP clients in llama.cpp:

| client | configured by | format | transport |
| --- | --- | --- | --- |
| llama-server | `--mcp-servers-config` | object with `mcpServers` | stdio only |
| llama.cpp Web UI | its MCP settings dialog | array of server objects | HTTP and SSE |

Set `BENTHIC_MCP_TRANSPORT=stdio` to serve over stdio, which is what llama-server's own client needs.
A cold spawn is about two seconds. `config/llama-mcp.README.md` has both config formats, the accepted
schema, and the two ways to fail silently; `config/llama-mcp-servers.example.json` and
`config/llama-mcp.example.json` are the two files.

For the HTTP service, which the Web UI uses and which needs a URL and a token because a browser cannot
spawn a subprocess:

```sh
mkdir -p ~/.config/benthic-mcp
cp config/benthic-mcp.env.example ~/.config/benthic-mcp/env
chmod 600 ~/.config/benthic-mcp/env
# Replace the token placeholder, and set MCP_HOST / LLAMA_HOST to your own hosts:
openssl rand -base64 36
```

```sh
scripts/start-benthic-mcp.sh
```

The service listens on port 8082 by default and exposes the Streamable HTTP MCP provider at
`http://<mcp-host>:8082/mcp`. `/health` requires the same bearer token.

`examples/` holds two starting points that need editing for your machine:
`start-llama-server.sh` is one tuned `llama-server` configuration (the one the numbers below were
measured under), and `benthic-mcp.service` is a user-level systemd unit.

**Set `enable_thinking=false` on the calling model.** It cuts completion tokens by about 64% and makes
the result reproducible, which is a deployment instruction for whatever calls the MCP rather than a
server setting. Its effect on the pass rate is about one case and is not established.
`docs/findings.md` records that asking the model in a system prompt not to deliberate does not work.

## The playbook

`benthic_playbook` returns dataset-specific instructions so the agent does not rediscover the schema
each session. Three modes, set by `BENTHIC_PLAYBOOK_MODE`:

- `off` - no playbook; the tool reports that it is disabled
- `seed` (default) - the curated seed in `src/benthic_mcp/seed.py`
- `active` - the promoted playbook, falling back to the seed if the file is missing

An agent can also report a mistake it made with `benthic_report`. **A reported lesson is never served
until it has been measured to help.** Accumulation used to promote on catalog verification alone,
which proves a statement is true and says nothing about whether it changes behaviour; twelve
accumulated lessons were later measured and none earned a place, while one made a case measurably
worse. Lessons are now attributed against the case they came from before they can be served, and a
regression is quarantined. The measurement, the dead ends, and the limits of the instrument are in
[`docs/findings.md`](docs/findings.md).

## What is actually verified, and how

The statistical A/B is no longer the gate. It could not be: the suite's minimum detectable effect is
7.81 cases against 4.5 cases of total headroom, so a gate built on it can only ever detect harm,
which is why it kept reporting that nothing worked. Classifying all 1534 stored case-runs found that
**2 of 33 cases have ever moved because server code changed**. Three things replaced it.

**Contracts.** 24 properties over the catalog and join subsystems, stated in the code's own
vocabulary, with no model and no network, in `tests/test_contracts_catalog.py`. Each is falsifiable,
and each one was: six arrived violated and are now fixed, so the file's `xfail` list is empty. A
contract that arrives already green is asserting nothing.

**Answer checking.** The truth grader (`eval/truth/grade.py`) grades every case, from an end-to-end
`run_eval.py` run through the real MCP server to a direct run, and it reads two families of evidence:
what the model said, and what the tool trace shows. The result-level and route-level checks -
`server_right_count`, `rpc_server_row_count`, `historical_relation_queried`, `required_route_used`
and the rest - are what let a canary go red on a server defect without a model in the loop, because a
server returning the wrong rows fails a check even when the model faithfully reports what it was
given. The canary's own contracts run through the same grader, so the model-free gate and the graded
runs are one instrument. Every check reports what it expected and what it found: "wrong" is not a
finding.

**A canary that was tested by breaking it.** `eval/canary/questions.json` is six cases: one per
signed join path, plus the one non-join case that expects rows.

| server | verdict |
| --- | --- |
| correct | **18/18 green** |
| `_type_coercion` forced off, reproducing the `'03'`/`3` bug | **8/12, red on exactly the two district cases** |

The four zero-row guards stayed green against the broken server, because silence satisfies a zero
expectation. That is the predicted behaviour and the reason the canary keeps cases in both
directions. It runs in **184s/rep against 778s/rep** for the full suite.

```sh
uv run python eval/run_eval.py --questions eval/canary/questions.json --reps 3
```

Report it as green or red, never as a percentage: six cases over four paths is a step function. The
30-case suite is tier 2, reported alongside it but not gating.

### What they found

Six real defects, none of which the model-in-the-loop apparatus saw:

- A signed join between a `string` column holding `'03'` and an `integer` column holding `3` compared
  them in Python, matched nothing, and returned 0 rows where 127 exist. 55 of 55 observed calls. The
  two affected cases scored as **passes** for 51 stored runs, and a `benthic_report` had diagnosed it
  correctly while the harness could not see it.
- The columns a `context_conditions` entry compares were never fetched, so no partial signed join
  could ever match a row, for any input - the unit test hand-built rows that already had the column.
- `number` was missing from the numeric type set. It is the catalog's own name for a decimal and the
  second most common declaration in the signed manifest, at 427 of 3419 columns.
- `order=` and `having` each computed a comparison the caller never asked for, on a column whose
  declared type PostgREST did not honour, and raised a `TypeError` the tool wrapper does not catch.
- Two of the suite's cases asked for a chain whose first hop is provably empty, and the scorer
  counted a truthful dead end as a failure. They now ask where the chain terminates, 4/4.
- "What is the largest award?" was unanswerable on exactly the large relations where it matters. The
  natural query - a `max` aggregate - was refused as a complete scan, and the fallback
  `order=col:desc limit=1` returned the NULLS FIRST, so a null-heavy column yielded `null` instead of
  the maximum. Both routes dead-ended, and the case passed one run in three. A `max` or `min` has one
  output row however wide the source is, so it is now fetched as `order=col.desc.nullslast limit=1` -
  one indexed page, no scan.

### What has been measured

33 generated cases against a 35B MoE coder model, 6-turn budget - measured when the suite held 33
cases; it now holds 30, and the counts below are the record of that run rather than a current figure.
Read this as a count of cases answered correctly, not as a measurement of anything: the suite's
minimum detectable effect is larger than the whole remaining failure set, so a difference of one or
two cases between rows here is not distinguishable from noise.

| configuration | result |
| --- | --- |
| seed, `enable_thinking=false`, 2 reps | **32/33 both reps**, holdout 8/8 and 7/8 |
| completion tokens, thinking off | 19,092 / 20,226 per rep, against 53,231 / 56,917 with it |

`enable_thinking=false` cuts completion tokens by about 64%, and nothing in the run-to-run spread
comes close to that. Its effect on the pass rate is not claimed, because this suite cannot resolve an
effect that size. `docs/findings.md` records the arithmetic and the three measurements that got there.

The honest position on self-improvement is worth stating plainly: **the loop does not learn.** Its
8 recorded lessons were never measured - `source_ref` was added after all of them were written, so
the attribution gate had nothing to read and had run zero times. Provenance has been reconstructed
from the round log, and 6 lessons are measurable for the first time. The loop also has no way to
express a code change, which is the only kind that has ever worked here - five of five kept
interventions were code, four of four reverted ones were prose. The mechanism that works is
contracts plus a reviewer.

### Graded truth, and what it caught that nothing else did

The suites above grade against expectations written by whoever ran the tool last, which is how a
suite becomes a rubber stamp. `eval/truth/` derives its numbers from the database itself - four
kinds of case, **27 cases over 15 relations**, every value cross-checked with `psql` before it was
committed. Grading is per capability and reported per capability; there is no aggregate rate, because
an average over exact RPCs beside heuristic joins is a number nobody can act on.

The first full run of that corpus measured **16/21**, and the five failures were all "what is the
largest X" on relations over the complete-scan limit. No unit test could have seen it: the defect
exists only on the large relations, and the corpus had never covered one. After the fix, the same
corpus passes **27/27**, and a query-light session costs about a third of its prompt tokens less than
it did, because the discover payload now spells null as absence - 56.6% of its bytes were null
spellings of `description`, `unit` and `srid`, which carry no information an omitted key does not.

The truth corpus is hand-targeted; `eval/coverage/` is its broad counterpart. It holds **1012 cases
over 87 of the 99 queryable relations**, generated from the signed manifest so a relation or column
nobody thought to target still has a case. Five kinds of case: row counts, non-null counts, smallest
ordered values, filtered maxima, and grouped maxima. Every value is derived through the signed
PostgREST endpoint - never through the server under test, because a number fetched through the server
inherits every defect the case exists to catch - and every case records the request it was derived
from, the manifest hash it was signed under, and its tolerances. The corpus is measured by slice;
results and the failures each slice produced are in `eval/coverage/RESULTS.md`.

## Evaluation

```sh
# Tier 1, the gate: six cases, one per signed join path, 184s/rep
uv run python eval/run_eval.py --questions eval/canary/questions.json --reps 3

# Tier 2, report only: the full 30 cases. Measures turn discipline more than server correctness
uv run python eval/run_eval.py                       # run it
uv run python eval/generate_cases.py --output-dir eval/generated-tmp  # rebuild elsewhere; eval/generated is pinned by the canary
uv run python eval/run_eval.py --case-filter multi_step --reps 5
```

Each run writes the cases, model transcript, tool arguments and results, token usage, timings, and a
report under `eval/runs/<run-id>/`. Expected values come from direct signed PostgREST and RPC probes,
so the evaluator does not depend on the MCP query implementation it is scoring.

Four hand-verified cases in `eval/golden/questions.json` must pass with the seed playbook alone. They
are a check on the harness, the scorer, and the tool surface, so a failure there is a bug rather than a
result. `tests/test_golden.py` re-checks their expected values against the signed catalog, so a catalog
change cannot leave the suite quietly stale. They used to need `--max-turns 6` while the generated
suite ran at 5, because one case was failing about one run in five at five - a tripwire firing on a
capability it was not watching. The generated suite now defaults to 6 as well, so the special case is
gone; pass `--max-turns` explicitly if you are comparing against an older run.

Long runs belong in tmux so they can be watched:

```sh
scripts/tmux-run.sh my-run logs/run.log uv run python eval/run_eval.py
scripts/tmux-ls.sh
```

Two attribution tools sit above the suite, and they answer different questions:

```sh
# Does this one lesson fix the case it came from? Cheap, but biased against general advice.
uv run python eval/attrib.py --lesson-id <id> --case <case-id> --playbook eval/harness/cache/playbook.json

# Does this rule earn a place in the always-on core? Two full runs of the tuning split.
uv run python eval/attribute_suite.py --rule "Never end the turn without a final answer." --reps 1
```

### Observation, which is the primary instrument

The suites above score against expected values. What follows does not, and it is what actually shows
whether the model can use the tools, because it records the model's own behaviour rather than a
verdict about it.

`benthic-observe.timer` runs `scripts/observe-once.sh` every 30 minutes, which calls
`scripts/tick.sh --probe`. One cycle refuses to run at all if `src/` is dirty, if 8081 is unhealthy,
if fewer than six MCP tools are registered, or if the repository toplevel is one whose guidelines
forbid autonomous agents. Then it runs the contracts, probes, and surveys.

```sh
# One cycle by hand. The timer runs exactly this.
scripts/tick.sh --probe

# The probe set is eval/observer/probes/core.json; sweep.py records every turn including reasoning.
uv run python eval/observer/sweep.py \
  --probes eval/observer/probes/core.json \
  --record eval/observer/records/manual.jsonl \
  --max-turns 12

# Reads the server's own refusal text, not the model's prose.
uv run python eval/observer/findings.py --records eval/observer/records/*.jsonl
```

The cycle observes and reports. It never edits code or guidance, because deciding whether a failure is
the server's fault or the model's is judgement, and an unattended loop that guesses at causes
manufactures false greens at a rate no review catches.

One guard is worth knowing about. A case the server did not properly answer is an outage, not a model
failure, and it is recorded as `invalid` rather than as a failure. Two shapes count: no turns at all
with the elapsed time at the tool timeout, and a case where every call that came back was a 5xx. The
second shape matters because such a case *does* have turns, so a guard that only asks "did it get
turns" passes it and counts the outage as evidence about the model - which is how 429 outage errors
sat in `refusal_kinds` for six days and read as a live defect that did not exist. One genuine refusal
among the 5xx calls means the server answered something, so the case counts as the model's again.
Measured over the recorded history, 288 of those 5xx responses sit in 45 cases, all from a single day.

`findings.py` refuses to write findings at all if the newest cycle contains an invalid case, and the
survey skips invalid cases so their errors cannot reach the counters. Without any of this, a server
that stopped answering generation still served `/health` and `/props`, and the cycle reported the
outage as a pass rate: across this repository's recorded history, 459 of 1,527 cases were server
outages.

### The combination suite

`eval/combination/` runs 100 real multi-hop questions against the live model, written by the operator
and held in `grok/` because it is theirs rather than ours.

```sh
# Roughly 8 hours for 100 cases. Belongs in tmux.
scripts/tmux-run.sh suite logs/suite.log \
  uv run python eval/combination/classify.py --out eval/combination/run-$(date +%Y%m%dT%H%M) \
  --max-turns 12

# Reads refusals for ones that hand back the method they just refused.
uv run python eval/combination/refusal_shape.py --cases eval/combination/run-*/cases.jsonl
```

The suite carries no expected values, so `classify.py` refuses to score it and reports outcomes rather
than grades. "Answered" means the model produced an answer, not a right one. Results and what they do
not establish are in [`eval/combination/RESULTS.md`](eval/combination/RESULTS.md).

## Configuration

| Variable | Default | Purpose |
| --- | --- | --- |
| `BENTHIC_BDP_ROOT` | `https://benthic.io/bdp` | Signed BDP document root |
| `BENTHIC_COLLECTIONS` | `ngopen` | Signed collections to load |
| `BENTHIC_TRUSTED_KEYS` | Current pinned key | Trusted Ed25519 public keys |
| `BENTHIC_CACHE_DIR` | `$XDG_CACHE_HOME/benthic-mcp` | Verified catalog cache |
| `BENTHIC_CACHE_TTL_SECONDS` | `900` | Refresh interval |
| `BENTHIC_MAX_CACHE_AGE_SECONDS` | `604800` | Maximum stale-cache fallback age |
| `BENTHIC_REQUEST_TIMEOUT_SECONDS` | `30` | Upstream request timeout |
| `BENTHIC_MAX_ROWS` | `1000` | Maximum PostgREST page size |
| `BENTHIC_DEFAULT_QUERY_LIMIT` | `100` | Default result limit |
| `BENTHIC_AGGREGATE_SCAN_LIMIT` | `10000` | Maximum complete source scan |
| `BENTHIC_MAX_RESPONSE_BYTES` | `1048576` | Per-response byte limit |
| `BENTHIC_MCP_HOST` | `0.0.0.0` | HTTP bind address |
| `BENTHIC_MCP_PORT` | `8082` | HTTP port |
| `BENTHIC_MCP_PATH` | `/mcp` | Streamable HTTP path |
| `BENTHIC_MCP_ALLOWED_HOSTS` | loopback only | Accepted Host headers |
| `BENTHIC_MCP_ALLOWED_ORIGINS` | loopback only | Accepted browser origins |
| `BENTHIC_MCP_TRANSPORT` | `http` | `http` or `stdio`; stdio is what llama-server's MCP client needs |
| `BENTHIC_MCP_BEARER_TOKEN` | unset | Required for the HTTP transport only |
| `BENTHIC_PLAYBOOK_MODE` | `seed` | `off`, `seed`, or `active` |
| `BENTHIC_PLAYBOOK_PATH` | `<cache>/playbook.json` | Promoted playbook location |
| `BENTHIC_PLAYBOOK_TOKEN_BUDGET` | `600` | Token cap for the always-on core slice |
| `BENTHIC_TRACE_ENABLED` | `1` | Record objective tool-call traces |
| `BENTHIC_TRACE_RETENTION_DAYS` | `30` | Trace retention |
| `BENTHIC_LESSON_RETENTION_DAYS` | `30` | Lesson retention |
| `BENTHIC_TRACE_INCLUDE_TEXT` | `0` | Store question text from `benthic_report` |
| `BENTHIC_CONSOLIDATOR_LLM_URL` | local llama-server | Model used by the consolidator |

The shipped allow-lists name loopback only, deliberately: they gate the `Host` and `Origin` headers,
so a default that named a particular machine would silently authorise that host for anyone who
installed the server without reading the configuration.

The fetched `keys.json` is not used as a trust root. Add rotated trusted keys through
`BENTHIC_TRUSTED_KEYS` only after out-of-band verification.

The API does not expose server-side grouped aggregates. A group-and-sum over a relation whose rows
cannot all be fetched is refused, and the refusal names the real count rather than returning partial
totals: narrowing to a period, or using a signed pre-aggregated relation, are the routes. A `max`,
`min`, `count` or `count(col)` has one output row however wide the source is, so those are answered
from a single indexed page and are not subject to the complete-scan limit - see
[What the server will and will not answer](#what-the-server-will-and-will-not-answer).

If the signed catalog changes after a playbook was generated, the fingerprint no longer matches and the
server serves the static core plus a staleness notice instead of stale dataset detail, until the
candidate is re-consolidated and re-gated.

## Development

```sh
uv run ruff check .
uv run ruff format --check .
uv run pyright
uv run pytest -m "not live"
```

The non-live marker excludes tests that need a real BDP endpoint and a running `llama-server`:

```sh
BENTHIC_LIVE_TESTS=1 uv run pytest -m live
BENTHIC_LIVE_TESTS=1 BENTHIC_LIVE_REGRESSION=1 uv run pytest -m live
```

`tests/test_loop_invariants.py` is worth reading before changing the improvement loop. It states the
properties that make a self-modifying system trustworthy - nothing is served without a measured
effect, a verdict is auditable, the document cannot self-perpetuate - rather than examples of them.

Long-running changes to the playbook belong in a disposable sandbox with `BENTHIC_CACHE_DIR` pointed
somewhere that is not the live store, so a run never serves a candidate:

```sh
BENTHIC_CACHE_DIR=eval/harness/cache uv run python eval/harness.py --rounds 6
```

## Documentation

- [`MEMORY.md`](MEMORY.md) - working memory: what is deployed, what has been measured, what is still
  open, and the failure modes this project keeps hitting
- [`docs/findings.md`](docs/findings.md) - what the harness measured, what did not work, and the limits
  of the measurement
- [`eval/truth/`](eval/truth/) - the derived-truth suites: how a numeric expectation is derived from
  the database, cross-checked with `psql`, and graded per capability. Read `grade.py`'s module
  docstring before changing a check; it states what the grader will and will not claim
- [`eval/combination/RESULTS.md`](eval/combination/RESULTS.md) - the 100-case combination suite, its
  failure taxonomy, and the corrections made to earlier readings of it
- [`docs/opencode-automation.md`](docs/opencode-automation.md) - driving an OpenCode agent from a shell
  timer, with what was verified by running it and what was only undocumented
- [`SECURITY.md`](SECURITY.md) - reporting a vulnerability, deployment notes, and the trust boundary

## License

MIT. See [`LICENSE`](LICENSE).