You can explore, query, join, and spatially analyze signed Benthic datasets through a read-only, manifest-verified MCP server.
Discover relevant signed relations, columns, join paths, and reliability warnings with benthic_discover.
Fetch dataset access conventions and signed hop-by-hop routes between relations with benthic_playbook.
Run bounded single-relation queries with filters, ordering, grouping, aggregates, and having clauses via benthic_query.
Join multiple relations only along signed BDP join paths with benthic_join.
Call three allowlisted spatial RPCs: find_district, districts_in_bbox, and nonprofits_nearby.
Report one mistake and its correction as a pending lesson with benthic_report; lessons are not served until verified and A/B measured.
The server refuses unsafe or unanswerable requests: arbitrary URLs, relations, SQL, writes, invented joins, and aggregates over sources that cannot be fully scanned or deterministically paged.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@benthic-mcp find signed datasets about ocean acidification"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A read-only MCP server over signed Benthic Data Provenance datasets, plus the evaluation harness that measures whether the playbook it serves actually improves an agent's answers.
The server exposes six tools:
tool | what it does |
| finds the smallest relevant signed relation, its columns, and its join paths |
| returns dataset access conventions, or the signed route between two relations |
| runs a bounded single-relation query with compact filter, aggregate, and |
| runs a multi-relation plan using only signed BDP join paths |
| runs three allowlisted spatial RPCs |
| records one mistake and its correction, held pending until verified and A/B measured |
Plain-language questions are interpreted by the calling chat model, which turns a question into compact tool arguments. This service validates and executes those arguments without embedding a model of its own, so what an agent is allowed to ask is decided by a signed manifest rather than by a prompt.
The interesting part of this server is not what it serves but where it stops, and the refusal is designed to be terminal rather than a hint the model talks its way past. A question that cannot be answered from the signed catalog gets "this cannot be answered" and an instruction to say so - not a suggestion to narrow filters that cannot help.
A max or min is a top-1, not a scan. One indexed page, order=col.desc.nullslast limit=1.
The complete-scan limit does not apply to it, because it has one output row however wide the source
is. .nullslast matters: Postgres sorts NULLS FIRST on a descending sort, so without it a
null-heavy column answers null instead of the extreme.
A bare count(*) and a count(col) are HEAD counts. Both have one output row, so neither is
refused for size. count(col) sends col=not.is.null on the same HEAD, which is exactly the
non-null count a scan would compute.
An aggregate over a source that cannot be read in full is refused, and the message names the real count that makes it impossible rather than telling the caller to narrow.
A relation with no primary key cannot be paged deterministically, so an aggregate over one is refused however far it is narrowed - getting under the scan limit does not help, and no sequence of queries reaches a cross-row total there.
A complete join result of identical rows says so. One left row matching every right row is a
fan-out, and the result carries the representative row plus the exact row_count, so a complete
result cannot read as a truncated page.
Discovery payload omits nulls. Absence and null say the same thing, and the payload is
re-sent to the model on every remaining turn.
Every refusal was written because the alternative - a wrong number that reads as a result - is worse than a clean stop.
Related MCP server: data-agent-service
The catalog is signed. The server pins an Ed25519 authority key, verifies RFC 8785 canonical JSON, SHA-256 payload hashes, and every member manifest against the collection hash, and then serves only what the signature covers:
only relations marked queryable: true in a signed manifest
only join paths present in the signed join graph
only an allowlist of RPC endpoints, hand-written in src/benthic_mcp/catalog.py
no arbitrary URLs, relations, SQL, or write operations
API and RPC traffic restricted to HTTPS endpoints on benthic.io
The playbook this server serves to the model is prose, and prose is model-authored. It can therefore
never widen any of the above. A fact the signed catalog already carries is always read from the
verified manifest at serve time, so guidance cannot contradict it. See
SECURITY.md for the trust boundary in one page.
Python 3.12 or newer
Network access to https://benthic.io
A local llama-server for the agent that calls the tools
git clone https://github.com/benthic-io/benthic-mcp
cd benthic-mcp
uv sync --frozenTwo transports are supported, and they are not interchangeable with the two MCP clients in llama.cpp:
client | configured by | format | transport |
llama-server |
| object with | stdio only |
llama.cpp Web UI | its MCP settings dialog | array of server objects | HTTP and SSE |
Set BENTHIC_MCP_TRANSPORT=stdio to serve over stdio, which is what llama-server's own client needs.
A cold spawn is about two seconds. config/llama-mcp.README.md has both config formats, the accepted
schema, and the two ways to fail silently; config/llama-mcp-servers.example.json and
config/llama-mcp.example.json are the two files.
For the HTTP service, which the Web UI uses and which needs a URL and a token because a browser cannot spawn a subprocess:
mkdir -p ~/.config/benthic-mcp
cp config/benthic-mcp.env.example ~/.config/benthic-mcp/env
chmod 600 ~/.config/benthic-mcp/env
# Replace the token placeholder, and set MCP_HOST / LLAMA_HOST to your own hosts:
openssl rand -base64 36scripts/start-benthic-mcp.shThe service listens on port 8082 by default and exposes the Streamable HTTP MCP provider at
http://<mcp-host>:8082/mcp. /health requires the same bearer token.
examples/ holds two starting points that need editing for your machine:
start-llama-server.sh is one tuned llama-server configuration (the one the numbers below were
measured under), and benthic-mcp.service is a user-level systemd unit.
Set enable_thinking=false on the calling model. It cuts completion tokens by about 64% and makes
the result reproducible, which is a deployment instruction for whatever calls the MCP rather than a
server setting. Its effect on the pass rate is about one case and is not established.
docs/findings.md records that asking the model in a system prompt not to deliberate does not work.
benthic_playbook returns dataset-specific instructions so the agent does not rediscover the schema
each session. Three modes, set by BENTHIC_PLAYBOOK_MODE:
off - no playbook; the tool reports that it is disabled
seed (default) - the curated seed in src/benthic_mcp/seed.py
active - the promoted playbook, falling back to the seed if the file is missing
An agent can also report a mistake it made with benthic_report. A reported lesson is never served
until it has been measured to help. Accumulation used to promote on catalog verification alone,
which proves a statement is true and says nothing about whether it changes behaviour; twelve
accumulated lessons were later measured and none earned a place, while one made a case measurably
worse. Lessons are now attributed against the case they came from before they can be served, and a
regression is quarantined. The measurement, the dead ends, and the limits of the instrument are in
docs/findings.md.
The statistical A/B is no longer the gate. It could not be: the suite's minimum detectable effect is 7.81 cases against 4.5 cases of total headroom, so a gate built on it can only ever detect harm, which is why it kept reporting that nothing worked. Classifying all 1534 stored case-runs found that 2 of 33 cases have ever moved because server code changed. Three things replaced it.
Contracts. 24 properties over the catalog and join subsystems, stated in the code's own
vocabulary, with no model and no network, in tests/test_contracts_catalog.py. Each is falsifiable,
and each one was: six arrived violated and are now fixed, so the file's xfail list is empty. A
contract that arrives already green is asserting nothing.
Answer checking. The truth grader (eval/truth/grade.py) grades every case, from an end-to-end
run_eval.py run through the real MCP server to a direct run, and it reads two families of evidence:
what the model said, and what the tool trace shows. The result-level and route-level checks -
server_right_count, rpc_server_row_count, historical_relation_queried, required_route_used
and the rest - are what let a canary go red on a server defect without a model in the loop, because a
server returning the wrong rows fails a check even when the model faithfully reports what it was
given. The canary's own contracts run through the same grader, so the model-free gate and the graded
runs are one instrument. Every check reports what it expected and what it found: "wrong" is not a
finding.
A canary that was tested by breaking it. eval/canary/questions.json is six cases: one per
signed join path, plus the one non-join case that expects rows.
server | verdict |
correct | 18/18 green |
| 8/12, red on exactly the two district cases |
The four zero-row guards stayed green against the broken server, because silence satisfies a zero expectation. That is the predicted behaviour and the reason the canary keeps cases in both directions. It runs in 184s/rep against 778s/rep for the full suite.
uv run python eval/run_eval.py --questions eval/canary/questions.json --reps 3Report it as green or red, never as a percentage: six cases over four paths is a step function. The 30-case suite is tier 2, reported alongside it but not gating.
Six real defects, none of which the model-in-the-loop apparatus saw:
A signed join between a string column holding '03' and an integer column holding 3 compared
them in Python, matched nothing, and returned 0 rows where 127 exist. 55 of 55 observed calls. The
two affected cases scored as passes for 51 stored runs, and a benthic_report had diagnosed it
correctly while the harness could not see it.
The columns a context_conditions entry compares were never fetched, so no partial signed join
could ever match a row, for any input - the unit test hand-built rows that already had the column.
number was missing from the numeric type set. It is the catalog's own name for a decimal and the
second most common declaration in the signed manifest, at 427 of 3419 columns.
order= and having each computed a comparison the caller never asked for, on a column whose
declared type PostgREST did not honour, and raised a TypeError the tool wrapper does not catch.
Two of the suite's cases asked for a chain whose first hop is provably empty, and the scorer counted a truthful dead end as a failure. They now ask where the chain terminates, 4/4.
"What is the largest award?" was unanswerable on exactly the large relations where it matters. The
natural query - a max aggregate - was refused as a complete scan, and the fallback
order=col:desc limit=1 returned the NULLS FIRST, so a null-heavy column yielded null instead of
the maximum. Both routes dead-ended, and the case passed one run in three. A max or min has one
output row however wide the source is, so it is now fetched as order=col.desc.nullslast limit=1 -
one indexed page, no scan.
33 generated cases against a 35B MoE coder model, 6-turn budget - measured when the suite held 33 cases; it now holds 30, and the counts below are the record of that run rather than a current figure. Read this as a count of cases answered correctly, not as a measurement of anything: the suite's minimum detectable effect is larger than the whole remaining failure set, so a difference of one or two cases between rows here is not distinguishable from noise.
configuration | result |
seed, | 32/33 both reps, holdout 8/8 and 7/8 |
completion tokens, thinking off | 19,092 / 20,226 per rep, against 53,231 / 56,917 with it |
enable_thinking=false cuts completion tokens by about 64%, and nothing in the run-to-run spread
comes close to that. Its effect on the pass rate is not claimed, because this suite cannot resolve an
effect that size. docs/findings.md records the arithmetic and the three measurements that got there.
The honest position on self-improvement is worth stating plainly: the loop does not learn. Its
8 recorded lessons were never measured - source_ref was added after all of them were written, so
the attribution gate had nothing to read and had run zero times. Provenance has been reconstructed
from the round log, and 6 lessons are measurable for the first time. The loop also has no way to
express a code change, which is the only kind that has ever worked here - five of five kept
interventions were code, four of four reverted ones were prose. The mechanism that works is
contracts plus a reviewer.
The suites above grade against expectations written by whoever ran the tool last, which is how a
suite becomes a rubber stamp. eval/truth/ derives its numbers from the database itself - four
kinds of case, 27 cases over 15 relations, every value cross-checked with psql before it was
committed. Grading is per capability and reported per capability; there is no aggregate rate, because
an average over exact RPCs beside heuristic joins is a number nobody can act on.
The first full run of that corpus measured 16/21, and the five failures were all "what is the
largest X" on relations over the complete-scan limit. No unit test could have seen it: the defect
exists only on the large relations, and the corpus had never covered one. After the fix, the same
corpus passes 27/27, and a query-light session costs about a third of its prompt tokens less than
it did, because the discover payload now spells null as absence - 56.6% of its bytes were null
spellings of description, unit and srid, which carry no information an omitted key does not.
The truth corpus is hand-targeted; eval/coverage/ is its broad counterpart. It holds 1012 cases
over 87 of the 99 queryable relations, generated from the signed manifest so a relation or column
nobody thought to target still has a case. Five kinds of case: row counts, non-null counts, smallest
ordered values, filtered maxima, and grouped maxima. Every value is derived through the signed
PostgREST endpoint - never through the server under test, because a number fetched through the server
inherits every defect the case exists to catch - and every case records the request it was derived
from, the manifest hash it was signed under, and its tolerances. The corpus is measured by slice;
results and the failures each slice produced are in eval/coverage/RESULTS.md.
# Tier 1, the gate: six cases, one per signed join path, 184s/rep
uv run python eval/run_eval.py --questions eval/canary/questions.json --reps 3
# Tier 2, report only: the full 30 cases. Measures turn discipline more than server correctness
uv run python eval/run_eval.py # run it
uv run python eval/generate_cases.py --output-dir eval/generated-tmp # rebuild elsewhere; eval/generated is pinned by the canary
uv run python eval/run_eval.py --case-filter multi_step --reps 5Each run writes the cases, model transcript, tool arguments and results, token usage, timings, and a
report under eval/runs/<run-id>/. Expected values come from direct signed PostgREST and RPC probes,
so the evaluator does not depend on the MCP query implementation it is scoring.
Four hand-verified cases in eval/golden/questions.json must pass with the seed playbook alone. They
are a check on the harness, the scorer, and the tool surface, so a failure there is a bug rather than a
result. tests/test_golden.py re-checks their expected values against the signed catalog, so a catalog
change cannot leave the suite quietly stale. They used to need --max-turns 6 while the generated
suite ran at 5, because one case was failing about one run in five at five - a tripwire firing on a
capability it was not watching. The generated suite now defaults to 6 as well, so the special case is
gone; pass --max-turns explicitly if you are comparing against an older run.
Long runs belong in tmux so they can be watched:
scripts/tmux-run.sh my-run logs/run.log uv run python eval/run_eval.py
scripts/tmux-ls.shTwo attribution tools sit above the suite, and they answer different questions:
# Does this one lesson fix the case it came from? Cheap, but biased against general advice.
uv run python eval/attrib.py --lesson-id <id> --case <case-id> --playbook eval/harness/cache/playbook.json
# Does this rule earn a place in the always-on core? Two full runs of the tuning split.
uv run python eval/attribute_suite.py --rule "Never end the turn without a final answer." --reps 1The suites above score against expected values. What follows does not, and it is what actually shows whether the model can use the tools, because it records the model's own behaviour rather than a verdict about it.
benthic-observe.timer runs scripts/observe-once.sh every 30 minutes, which calls
scripts/tick.sh --probe. One cycle refuses to run at all if src/ is dirty, if 8081 is unhealthy,
if fewer than six MCP tools are registered, or if the repository toplevel is one whose guidelines
forbid autonomous agents. Then it runs the contracts, probes, and surveys.
# One cycle by hand. The timer runs exactly this.
scripts/tick.sh --probe
# The probe set is eval/observer/probes/core.json; sweep.py records every turn including reasoning.
uv run python eval/observer/sweep.py \
--probes eval/observer/probes/core.json \
--record eval/observer/records/manual.jsonl \
--max-turns 12
# Reads the server's own refusal text, not the model's prose.
uv run python eval/observer/findings.py --records eval/observer/records/*.jsonlThe cycle observes and reports. It never edits code or guidance, because deciding whether a failure is the server's fault or the model's is judgement, and an unattended loop that guesses at causes manufactures false greens at a rate no review catches.
One guard is worth knowing about. A case the server did not properly answer is an outage, not a model
failure, and it is recorded as invalid rather than as a failure. Two shapes count: no turns at all
with the elapsed time at the tool timeout, and a case where every call that came back was a 5xx. The
second shape matters because such a case does have turns, so a guard that only asks "did it get
turns" passes it and counts the outage as evidence about the model - which is how 429 outage errors
sat in refusal_kinds for six days and read as a live defect that did not exist. One genuine refusal
among the 5xx calls means the server answered something, so the case counts as the model's again.
Measured over the recorded history, 288 of those 5xx responses sit in 45 cases, all from a single day.
findings.py refuses to write findings at all if the newest cycle contains an invalid case, and the
survey skips invalid cases so their errors cannot reach the counters. Without any of this, a server
that stopped answering generation still served /health and /props, and the cycle reported the
outage as a pass rate: across this repository's recorded history, 459 of 1,527 cases were server
outages.
eval/combination/ runs 100 real multi-hop questions against the live model, written by the operator
and held in grok/ because it is theirs rather than ours.
# Roughly 8 hours for 100 cases. Belongs in tmux.
scripts/tmux-run.sh suite logs/suite.log \
uv run python eval/combination/classify.py --out eval/combination/run-$(date +%Y%m%dT%H%M) \
--max-turns 12
# Reads refusals for ones that hand back the method they just refused.
uv run python eval/combination/refusal_shape.py --cases eval/combination/run-*/cases.jsonlThe suite carries no expected values, so classify.py refuses to score it and reports outcomes rather
than grades. "Answered" means the model produced an answer, not a right one. Results and what they do
not establish are in eval/combination/RESULTS.md.
Variable | Default | Purpose |
|
| Signed BDP document root |
|
| Signed collections to load |
| Current pinned key | Trusted Ed25519 public keys |
|
| Verified catalog cache |
|
| Refresh interval |
|
| Maximum stale-cache fallback age |
|
| Upstream request timeout |
|
| Maximum PostgREST page size |
|
| Default result limit |
|
| Maximum complete source scan |
|
| Per-response byte limit |
|
| HTTP bind address |
|
| HTTP port |
|
| Streamable HTTP path |
| loopback only | Accepted Host headers |
| loopback only | Accepted browser origins |
|
|
|
| unset | Required for the HTTP transport only |
|
|
|
|
| Promoted playbook location |
|
| Token cap for the always-on core slice |
|
| Record objective tool-call traces |
|
| Trace retention |
|
| Lesson retention |
|
| Store question text from |
| local llama-server | Model used by the consolidator |
The shipped allow-lists name loopback only, deliberately: they gate the Host and Origin headers,
so a default that named a particular machine would silently authorise that host for anyone who
installed the server without reading the configuration.
The fetched keys.json is not used as a trust root. Add rotated trusted keys through
BENTHIC_TRUSTED_KEYS only after out-of-band verification.
The API does not expose server-side grouped aggregates. A group-and-sum over a relation whose rows
cannot all be fetched is refused, and the refusal names the real count rather than returning partial
totals: narrowing to a period, or using a signed pre-aggregated relation, are the routes. A max,
min, count or count(col) has one output row however wide the source is, so those are answered
from a single indexed page and are not subject to the complete-scan limit - see
What the server will and will not answer.
If the signed catalog changes after a playbook was generated, the fingerprint no longer matches and the server serves the static core plus a staleness notice instead of stale dataset detail, until the candidate is re-consolidated and re-gated.
uv run ruff check .
uv run ruff format --check .
uv run pyright
uv run pytest -m "not live"The non-live marker excludes tests that need a real BDP endpoint and a running llama-server:
BENTHIC_LIVE_TESTS=1 uv run pytest -m live
BENTHIC_LIVE_TESTS=1 BENTHIC_LIVE_REGRESSION=1 uv run pytest -m livetests/test_loop_invariants.py is worth reading before changing the improvement loop. It states the
properties that make a self-modifying system trustworthy - nothing is served without a measured
effect, a verdict is auditable, the document cannot self-perpetuate - rather than examples of them.
Long-running changes to the playbook belong in a disposable sandbox with BENTHIC_CACHE_DIR pointed
somewhere that is not the live store, so a run never serves a candidate:
BENTHIC_CACHE_DIR=eval/harness/cache uv run python eval/harness.py --rounds 6MEMORY.md - working memory: what is deployed, what has been measured, what is still
open, and the failure modes this project keeps hitting
docs/findings.md - what the harness measured, what did not work, and the limits
of the measurement
eval/truth/ - the derived-truth suites: how a numeric expectation is derived from
the database, cross-checked with psql, and graded per capability. Read grade.py's module
docstring before changing a check; it states what the grader will and will not claim
eval/combination/RESULTS.md - the 100-case combination suite, its
failure taxonomy, and the corrections made to earlier readings of it
docs/opencode-automation.md - driving an OpenCode agent from a shell
timer, with what was verified by running it and what was only undocumented
SECURITY.md - reporting a vulnerability, deployment notes, and the trust boundary
MIT. See LICENSE.
Find the smallest relevant signed source, columns, and join paths for a question.
Use the returned source field verbatim in benthic_query. If join_paths is empty, make separate
benthic_query calls; never invent a join.
One call is normally enough: it returns the best relations together with their columns, and
columns_truncated says when that list is partial. Ask again for more columns only when
columns_truncated is true, by passing detail='full'. A wrong column guess is recoverable rather
than fatal, because benthic_query reports the near-miss signed name.Core access rules:
Call benthic_discover before benthic_query, and pass its source value verbatim.
Use benthic_join only for a signed path returned by discovery; never invent a join.
Partial joins require context_conditions and must be reported as provisional.
Heuristic matches are not exact; keep the returned reliability warning in the answer.
Never derive totals from truncated or incomplete source data.
Call benthic_playbook(dataset) when a question needs domain conventions for that dataset.
Dates in this catalog are ISO-8601; filter with gte/lte rather than string comparison.
Never use a current-only view to answer a historical question.
You have a limited number of turns, so once you have the rows you need you must stop calling tools and write the final answer, because a question left unanswered scores zero even when every call succeeded.
usaspending: Federal award transactions and the recipient entity index.
irs_ng: IRS exempt organization records, including Form 990 details.
samer: SAM entity registrations, the bridge between USAspending and IRS identifiers.
up_cdmaps: Congressional district boundaries, queried through spatial RPCs.
usp_cl: Congressional legislators and executive terms, including historical terms.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| query | No | ||
| detail | No | summary | |
| dataset | No | ||
| relation | No |
| Name | Required | Description |
|---|---|---|
| query | Yes | |
| warnings | No | |
| relations | Yes | |
| join_paths | Yes | |
| total_matches | Yes | |
| more_available | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover the read-only/idempotent safety profile, yet the description adds substantial behavior: what an empty join_paths means, that columns_truncated signals a partial list, how to escalate with detail='full', and that a wrong column guess is recoverable because benthic_query reports the near-miss name. It also discloses that partial joins are provisional and must be reported as such.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The workflow guidance is front-loaded and each access rule is actionable, but the text carries noticeable redundancy ('never invent a join' and the 'source verbatim' instruction each appear twice) and mixes family-wide rules into a single tool's definition. Adequately sized but not tight.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be documented, and the description still explains the key return signals (join_paths, columns_truncated). For a multi-step discovery tool it is largely complete, with only the limit/relation parameters left unaddressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does explain the detail parameter ('Ask again for more columns... by passing detail=full') and enumerates the dataset vocabulary (usaspending, irs_ng, samer, up_cdmaps, usp_cl), but limit, relation, and query are left without semantic explanation, so compensation is only partial.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Find the smallest relevant signed source, columns, and join paths for a question.' It also differentiates itself from siblings by naming when its output feeds benthic_query and benthic_join, so an agent can place it in the workflow without opening other schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit when-to-use and when-not rules: 'Call benthic_discover before benthic_query', 'If join_paths is empty, make separate benthic_query calls; never invent a join', and routing to benthic_playbook for domain conventions. It also gives a stopping condition for the whole workflow ('once you have the rows you need you must stop calling tools').
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run one signed join using qualified sources.
Omit left_column and right_column when exactly one reliable identifier path connects the two
sources, and the signed path is resolved for you. That is the common case and it saves having to
look the path up first. Columns are still required whenever the path is ambiguous, absent, or
anything other than a reliable identifier join, so a heuristic or partial join is never chosen
for you; the error lists the candidates instead.
The adapter follows signed reliability metadata and returns warnings. Partial joins require
context_conditions. The left key filter is propagated to the right source; do not invent columns
or unsigned relationships.Call benthic_playbook(dataset) first when a question depends on dataset conventions, then call benthic_report once if this tool call taught you something the playbook lacks.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | inner | |
| limit | No | ||
| question | Yes | ||
| left_where | No | ||
| left_column | No | ||
| left_source | Yes | ||
| right_where | No | ||
| right_column | No | ||
| right_source | Yes | ||
| context_conditions | No |
| Name | Required | Description |
|---|---|---|
| rows | Yes | |
| joins | Yes | |
| columns | Yes | |
| sources | Yes | |
| warnings | No | |
| row_count | Yes | |
| truncated | Yes | |
| next_offset | No | |
| source_complete | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, and the description adds meaningful behavior beyond them: it follows signed reliability metadata, returns warnings, propagates the left key filter to the right source, requires context_conditions for partial joins, and refuses to pick heuristic joins. This is useful disclosed behavior; it stops short of describing output shape but an output schema exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The core rule is front-loaded, which is good, but the prose is jargon-heavy ('signed', 'signed path', 'signed reliability metadata') and the trailing meta-instruction about benthic_playbook/benthic_report adds length. It is understandable but not tight.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, return values need not be explained, and safety is covered by annotations. The description resolves the main ambiguity (when columns are optional/required) and the error path. Some parameters and the meaning of 'mode' remain unexplained, keeping it from a 5.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 10 parameters, so the description carries the burden. It clarifies left_column/right_column omission rules, context_conditions, and left_where propagation, but says nothing about mode, limit, right_where, or question. Partial compensation only.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb+resource: 'Run one signed join using qualified sources.' This is clearly a join-execution tool distinct from the query/discover/rpc siblings. It does not explicitly name which sibling to use instead, so it misses the top tier.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a concrete decision rule: omit left_column/right_column when exactly one reliable identifier path exists, provide them when the path is ambiguous/absent. It also states the error behavior (lists candidates) and the prerequisite for benthic_playbook. No explicit when-not-to-use-this-tool-vs-siblings guidance, so not a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Dataset-specific access conventions built from signed metadata and prior sessions.
Call once per dataset before answering a question that depends on domain conventions: which
source to use, which columns are authoritative, which mistakes to avoid, and which signed join
paths and RPC operations exist. Join recipes come from the signed catalog and are never
invented here. Omit dataset for a routing index of every dataset.
To move data between two relations, pass from_relation and to_relation as 'dataset.relation'
instead of a dataset. That returns the signed route hop by hop, shortest first, with the exact
arguments for each benthic_join call, and nothing else. Do this rather than searching
benthic_discover for a join path: a search can be rephrased, so a request for a path that does
not exist comes back looking like a request that was worded wrongly.
| Name | Required | Description | Default |
|---|---|---|---|
| dataset | No | ||
| max_hops | No | ||
| to_relation | No | ||
| from_relation | No |
| Name | Required | Description |
|---|---|---|
| core | No | |
| path | No | |
| status | Yes | |
| datasets | No | |
| warnings | No | |
| generator | No | |
| collection | Yes | |
| generated_at | No | |
| collection_notes | No | |
| catalog_fingerprint | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld, non-destructive. The description adds genuinely new behavior: join recipes come from the signed catalog and are never invented here, the relation mode returns 'nothing else,' and results are ordered shortest-first with exact arguments for each benthic_join call. Only max_hops behavior and response shape are left unstated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three well-organized paragraphs, front-loaded with the core purpose and the call-frequency guidance. The explanation of why not to use search (rephrasing masks non-existent paths) is slightly verbose but earns its place by steering tool selection.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists so return values needn't be described, and the two main invocation modes (dataset lookup vs relation routing) are covered. The only shortfall is the undocumented max_hops parameter, minor for a read-only advisory tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for 4 parameters, so the description must carry the load; it explains dataset (omit for routing index) and the from_relation/to_relation 'dataset.relation' format plus their mode-switching effect. max_hops is never explained beyond the schema's min/max bounds, a minor gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb (call/return) and resource (dataset-specific access conventions: authoritative columns, join paths, RPC operations) rather than restating the name. It partially distinguishes itself from siblings benthic_join and benthic_discover by naming them as alternatives, though the underlying resource ('playbook') remains slightly abstract.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit when-to-use ('Call once per dataset before answering a question that depends on domain conventions'), when-not ('Omit dataset for a routing index'), and a named alternative with the condition that selects it ('Do this rather than searching benthic_discover for a join path'). The rationale for preferring this tool over discover is even given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run one bounded read-only relation query.
Call benthic_discover first and pass its source value verbatim. where uses column=operator.value,
metrics uses output_name=function:column, having uses output_name>value, and order uses
column:asc or column:desc. The name in a metric is the output column, not the source column.
Operators: eq, neq, gt, gte, lt, lte, like, ilike, in, is.null, not.is.null. Aggregates are complete
or rejected; never infer totals from a truncated source.Call benthic_playbook(dataset) first when a question depends on dataset conventions, then call benthic_report once if this tool call taught you something the playbook lacks.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| order | No | ||
| where | No | ||
| having | No | ||
| offset | No | ||
| select | No | ||
| source | Yes | ||
| metrics | No | Aggregates as 'output_name=function:column' or 'output_name:function:column', where output_name is the output column, not a source column. Functions: count, sum, avg, min, max. 'n=count:*' counts rows. | |
| group_by | No | ||
| question | Yes |
| Name | Required | Description |
|---|---|---|
| rows | Yes | |
| joins | Yes | |
| columns | Yes | |
| sources | Yes | |
| warnings | No | |
| row_count | Yes | |
| truncated | Yes | |
| next_offset | No | |
| source_complete | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint/idempotentHint/openWorldHint, so safety is covered. The description adds real behavior beyond that: the query is bounded, aggregates are 'complete or rejected' rather than partially returned, and the caller must never infer totals from a truncated source. Those are non-obvious failure semantics the schema cannot express.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded one-line purpose followed by dense but relevant syntax clauses, then a short trailing routing sentence. Nearly every clause earns its place, though the metrics output-name rule and the '=' vs ':' variant are duplicated in the schema's metrics description, and the prose is tightly packed rather than cleanly separated.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need no explanation, and annotations cover the safety profile. What remains for a 10-parameter, low-schema-coverage query tool — the custom filter/aggregate DSL, the required discovery prerequisite, and the aggregate-completeness guarantee — is all present, so an agent has enough to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 10% (just the metrics field), so the description carries the burden. It defines the mini-language for where (column=operator.value), metrics (output_name=function:column, with the output-vs-source-column warning), having (output_name>value), and order (column:asc/desc), plus the operator list. It does not explain select, group_by, limit, or offset semantics, so it compensates substantially but not fully.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb+resource: 'Run one bounded read-only relation query.' The scope word 'bounded' plus 'read-only' pins down exactly what class of operation this is, and the text immediately routes the agent to benthic_discover for the source value, which makes it distinguishable from the other benthic_* siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit sequencing is given: 'Call benthic_discover first and pass its source value verbatim,' plus 'Call benthic_playbook(dataset) first when a question depends on dataset conventions.' That is concrete when-to-use guidance tied to named alternatives. It stops short of saying when to prefer benthic_join or benthic_rpc over this tool, so it is not a full when/when-not rubric.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Record one mistake you made and the correction, so future sessions do not repeat it.
Call this once after answering, only if a tool call or a source convention misled you. Write the lesson as an instruction to your future self ("use X instead of Y"), not as a complaint. Reports are stored as pending and are only served to later sessions after catalog verification and an offline A/B evaluation, so a wrong report cannot change current behaviour. Do not call it when nothing went wrong.
| Name | Required | Description | Default |
|---|---|---|---|
| lesson | Yes | ||
| dataset | No | ||
| symptom | Yes | ||
| relation | No | ||
| confidence | No | medium | |
| question_summary | No |
| Name | Required | Description |
|---|---|---|
| status | Yes | |
| recorded | Yes | |
| warnings | No | |
| lesson_id | Yes | |
| review_hint | Yes | |
| dataset_slice | No | |
| similar_pending | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the annotations by disclosing that reports are stored as pending and only served after catalog verification and an offline A/B evaluation, so a bad report cannot affect current behavior. It also clarifies the required phrasing style (instruction to future self, not a complaint) and the call-once constraint, which matches the non-idempotent annotation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action, then the trigger, then the safety note, then the exclusion. Every sentence contributes, though the pending/verification explanation runs slightly long.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists so return values need no explanation, and the mutation intent is clear against the annotations. However, with 0% schema coverage the four undocumented parameters (dataset, relation, confidence, question_summary) leave an agent guessing about what to supply.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must carry parameter meaning. It clarifies 'lesson' (write as an instruction, not a complaint) implicitly covers 'symptom', but says nothing about dataset, relation, confidence, or question_summary, leaving four of six parameters undefined.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource: recording one mistake and its correction so future sessions avoid it. The purpose is unmistakably distinct from the sibling query/discover/playbook tools, and an agent can tell what it does without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit timing ('Call this once after answering'), an explicit trigger condition ('only if a tool call or a source convention misled you'), and an explicit exclusion ('Do not call it when nothing went wrong'). This is textbook when/when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Call benthic_playbook(dataset) first when a question depends on dataset conventions, then call benthic_report once if this tool call taught you something the playbook lacks.
| Name | Required | Description | Default |
|---|---|---|---|
| lat | No | ||
| lon | No | ||
| max_lat | No | ||
| max_lon | No | ||
| min_lat | No | ||
| min_lon | No | ||
| congress | No | ||
| operation | Yes | The allowlist. This is the single source of truth for the benthic_rpc operation enum. | |
| radius_meters | No |
| Name | Required | Description |
|---|---|---|
| rows | Yes | |
| metadata | Yes | |
| warnings | No | |
| row_count | Yes | |
| truncated | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly=true, idempotent=true, destructive=false, and openWorld=true, so the safety profile is covered. The description adds nothing beyond that about behavior: no indication of what the three operations return, no dead-ends, no guidance on the one-shot workflow contract it implies.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
It is short, but its single sentence is spent on orchestration advice for other tools rather than front-loading this tool's purpose. Brevity here reflects under-specification, not discipline – no sentence earns its place for someone trying to call benthic_rpc.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With nine parameters at 11% schema coverage and three distinct enum operations, the description needed to carry substantial load and instead omits everything about them. An output schema exists, so return values are excused, but operation and parameter meaning are entirely absent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Nine parameters, schema description coverage of 11% (only 'operation' is documented, and that with an unhelpful 'allowlist' note). The description mentions zero parameters or their semantics, so the geographic fields (lat/lon/min/max) and radius_meters remain unexplained in both places.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description never states what benthic_rpc does. It says nothing about the RPC operations (find_district, districts_in_bbox, nonprofits_nearby) or the geography/district/nonprofit domain the tool operates over, so an agent cannot tell the tool's purpose from reading it. This is a missing purpose, not a vague one.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives sequencing guidance for two other tools (call benthic_playbook before, benthic_report after), but never says when to use benthic_rpc rather than benthic_query, benthic_join, or benthic_discover. The routing information present concerns siblings' invocation, not this tool's selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0benthic_discoverbenthic_joinbenthic_playbookbenthic_querybenthic_reportbenthic_rpcScored across 6 tools
Each tool has a distinct role: discovery, query, join, RPC, playbook, and report. The boundaries between benthic_join and a potential join via benthic_query are implied but not explicitly contrasted in the descriptions; however, the extensive cross-references and rules make their intended uses clear.
All tools follow a consistent benthic_ prefix with a single-word snake_case verb or noun (discover, query, join, rpc, playbook, report). The pattern is predictable and readable.
Six tools are well-scoped for a database query and metadata system. Each tool appears necessary: discovery, querying, joining, RPC access, playbook guidance, and reporting. No obvious redundancy or missing core operation.
The set covers the full lifecycle for this domain: discovering sources, querying data, performing joins, calling RPCs, accessing dataset conventions, and reporting issues. No obvious missing operations for read-only data exploration and joining.
Provenance-first database for teams and agents: every value carries sources, rules and coverage.
Machine-native research commons for agent evidence, discovery, rooms, and bounded research quests.
Query your org's data in natural language — read-only MCP access to SQL, NoSQL, files & warehouses.
A public commons for agents to search and share reusable findings and open research questions.
