# mcp-grounding

The `verify_place` grounding tool from
[camunda-trial](https://github.com/alxnhfr-bit/camunda-trial), re-exposed as an
**MCP server** and measured against the same 91-post golden set, with the same
deterministic scorer, so the two protocols can be compared on numbers.

The server is the smaller half. The comparison is the point.

> **Headline:** across three runs of the parity arm, the MCP path scores a mean
> of **87.9%** against the Camunda connector's **86.8%** — a 1.1-point gap
> inside a 2.2-point run-to-run spread. On this task, moving the tool between
> protocols did not change what the model got right. What it changed was
> smaller, reproducible, and more interesting than the accuracy number.

---

## The numbers

Same rubric (hash-pinned identical), same golden set, same model
(`claude-sonnet-5`), same scorer — `eval/score.mjs`, copied byte-identical from
camunda-trial and never edited. No model anywhere in the judging loop.

| | Camunda connector | MCP server (parity arm) |
|---|---|---|
| **Accuracy** | 79/91 — **86.8%** | mean **87.9%**, range 86.8–89.0% (n=3) |
| In-rubric posts | 17/17 (100%) | 17/17 (100%) in all 3 runs |
| Held-out posts | 62/74 (83.8%) | 86.5% / 83.8% / 85.1% (runs 1–3) |
| **Grounding discipline** | 7/7 (100%) | **100% in every run** |
| Restraint (golden `no_signal`, zero calls) | 62/62 (100%) | 61/62 (98.4%) in all 3 runs |
| Errored rows | 0 | 0 |
| Output-format repairs needed | n/a (schema-enforced) | 0 of 91, every run |

Individual runs: 89.0%, 86.8%, 87.9%. Temperature is unset in both systems, so
both are non-deterministic; a single run cannot distinguish a real difference
from noise, which is why the parity arm was run three times. One of those runs
reproduced Camunda's 79/91 exactly.

**Read that as: no detectable difference in accuracy.** The gap is smaller than
the instrument's own resolution. Anyone reporting a single 89.0% run as "MCP
beats Camunda by 2.2 points" would be reporting noise.

The confusion matrices are near-identical too. The 86.8% MCP run differs from
the Camunda run by exactly one post traded between buckets.

## The one reproducible difference

MCP calls the tool on a post Camunda never touched, in **every single run**,
across every arm:

> `1ute6a4` — *"Has anyone visited Sedona, Arizona? Worth the trip?"* — golden
> `no_signal`

Camunda: no tool call, labelled `no_signal`, correct. MCP: verifies Sedona
(`found=true`) every time, and in 2 of 3 parity runs then labels it
`rising_destinations` — wrong.

Nothing in the prompt differs. Same rubric bytes, same model, same policy text.
The difference is where the tool *sits*: in MCP the tool definition travels in
the request payload on every turn, and that appears to make it marginally more
reachable. One post out of 91 is not a law of nature, but 5 runs out of 5 is not
noise either.

This is the shape of finding the project was built to catch — too small to move
the headline number, invisible without a per-call trace, and a real behavioural
difference between the two protocols.

## What the tool description turned out to be worth

The build brief assumed the tool description would matter a great deal in MCP,
because "in the Camunda version, the BPMN wiring decided when the tool fired."

That premise was wrong, and finding out was the useful part. The Camunda rubric
already contained a `## Tool policy: verify_place` section — the *system prompt*
decided when the tool fired, not the BPMN. The connector's own description was
one sentence that Camunda Modeler read and no model ever did.

So the real question is not "does a description matter" but "does the spec
survive being moved into one". Three arms, same instruction, only its location
changing. Arm C is a negative control where the policy exists nowhere at all.

| arm | policy lives in | accuracy (n=91) | grounding | calls on non-rising posts |
|---|---|---|---|---|
| **A** parity | system prompt | 86.8–89.0% (n=3) | 100% | 1 (all 3 runs) |
| **B** MCP-native | tool description | 86.8% (n=1) | 100% | 2 |
| **C** control | nowhere | 85.7% (n=1) | 100% | 2 |

The last column counts every tool call on a post whose golden label is not
`rising_destinations`. It is deliberately broader than the scorer's own
restraint metric, which looks only at golden `no_signal` posts and so would not
have seen arm B verifying `Viator` on a `competitor_mentions` post.

**Moving the policy changed nothing.** On the 31-post day-2 subset, arms A and B
agreed on every post — same label, same tool-call count, 31 for 31. At n=91 arm
B lands inside arm A's range. The spec survives the move intact.

**Removing the policy did not break grounding. It broke restraint.** Arm C still
grounded every `rising_destinations` prediction it made. What degraded was the
*"do not call it otherwise"* half: spurious calls doubled.

That split says the two halves of the policy are held up by different things.
The obligation to ground has a backstop — the rubric's `## Output` section still
asks for `groundedPlaces` "for each place verified with found=true", so the
requirement is implied by the output contract even with the policy deleted.
Restraint has no backstop, and it is the first thing to go.

**The failure the brief predicted never appeared.** Not once across five runs and
455 classifications did the model assign `rising_destinations` without verifying
first — including in the arm where no policy existed anywhere. On this rubric
the grounding obligation is over-determined.

Every description version and its measurement: [docs/TOOL_DESCRIPTIONS.md](docs/TOOL_DESCRIPTIONS.md).

## Where the design decisions were

### Error semantics: a place that doesn't exist is not an error

The contract carried over from the Java connector, and the reason the tests were
written first:

| condition | Java connector | MCP server |
|---|---|---|
| place found | verdict, `found=true` | `CallToolResult`, `structuredContent.found=true` |
| **place not found** | **verdict, `found=false`** | **`CallToolResult`, `isError` unset** |
| auth / quota / transport | `ConnectorException` → incident | **JSON-RPC error** → client rejects, row fails |

The distinction that matters is the last row. MCP offers two different "errors",
and the tempting one is wrong here. A tool result with `isError: true` is handed
back to the *model* as text — so an expired API key becomes something the model
can reason around, and it can still answer `rising_destinations` having verified
nothing. A JSON-RPC error never reaches the model at all; the client's
`callTool()` rejects and the eval records the row as errored, which is exactly
where a Camunda incident put it.

**The SDK makes the right shape hard to reach.** `McpServer.registerTool()`
catches every error a tool callback throws and rewrites it into
`{ isError: true }`, discarding `error.code` and `error.data` — only the message
survives. A protocol-level failure is unreachable from a tool callback. This
server is therefore built on the low-level `Server` class, where the error code
survives to the client. See [FRICTION.md](FRICTION.md).

Asserted end-to-end over a real transport in
[`test/error-semantics.test.ts`](test/error-semantics.test.ts), including the
behavioural claim: *an infrastructure failure never reaches the model as a tool
result.*

### Which primitive is each thing

Decided by a type signature rather than by taste:

- **`verify_place` is a tool.** It performs an action with a side effect — an
  outbound call spending someone's quota — and the model chooses when.
- **The rubric is a resource** (`grounding://rubric/v3`), *not* a prompt. MCP's
  `PromptMessage` role is `"user" | "assistant"` — there is no system role. The
  rubric is a system prompt; returning it as a `user` message would move 13,688
  characters of instruction out of the system position and quietly change the
  experiment. A resource carries text with no role attached and lets the client
  place it.
- **`classify_post` is a prompt.** The hand-driving entry point for Claude Code.
  It carries the post, not the rubric — and *why* it cannot carry the rubric is
  the finding above.

### The credential is not a tool input

The Java connector took `apiKey` as an input property bound to
`{{secrets.GOOGLE_PLACES_API_KEY}}`, so it appeared as a field on the element
template. Here it is server configuration read from the environment and is
absent from the tool schema — the model can neither see it nor be talked into
supplying one. A test asserts no credential appears in the published schema.

## What "grounded" is actually worth

Worth stating plainly, because the project exists to avoid shipping plausible
verifications nobody checked — and the verification layer has its own false
positives.

Google Places Text Search is a *search*, not an existence check. It matches on
substrings and business names:

| query | verdict |
|---|---|
| `Zzyzxqwv Fictional Republic` | `found=true` → **Zzyzx, California**, a real road |
| `Atlantis` | `found=true` → a shop in Berlin |
| `Wakanda` | `found=false` |

So `found: true` means *"Places returned a best match for this query string"*,
not *"this place exists"*.

This is not hypothetical. In arm B the model verified **`Viator`** — a tour
booking platform, not a place — and Google returned `found=true`.

The tool is left unchanged: tightening it would diverge from the Java connector
and break the only thing making the two numbers comparable, and the golden set
never exercises the false-positive path. But it bounds the claim. "Perfect tool
discipline" means the tool was *called*, never that the place was real.

`eval/live-check.ts` reports these cases explicitly rather than hiding them.

## Running it

```bash
npm install
```

```bash
cp .env.example .env
```

Fill in `GOOGLE_PLACES_API_KEY` (Places API **New** enabled) and
`ANTHROPIC_API_KEY`. `.env` is gitignored.

```bash
npm test
```

Day-1 check against the live API — confirms a nonexistent place returns a
verdict, not an error:

```bash
node --env-file=.env eval/live-check.ts
```

A full 91-post run, then score it with the untouched scorer:

```bash
node --env-file=.env eval/run.ts --arm full --description v1-parity --label myrun
```

```bash
node eval/score.mjs eval/results/run-<timestamp>-myrun/results.jsonl
```

Compare finished runs (this calls `score.mjs`, it does not compute accuracy
itself):

```bash
node eval/summarise.mjs
```

### Attaching to Claude Code

`.mcp.json` is committed and loads the key via `--env-file`, so no secret enters
a tracked file. Claude Code reads it at startup.

## How it is put together

```
src/server/server.ts        MCP server on the low-level Server class
src/server/tool-description.ts  versioned descriptions, append-only
src/server/rubric.ts        both arms, split from one source so they cannot drift
src/places/client.ts        port of PlacesClient.java, same field mask, same codes
eval/run.ts                 MCP client + Messages API; tool def pulled from tools/list
eval/classify.ts            the agent loop, extracted so it can be tested
eval/score.mjs              copied byte-identical from camunda-trial. Never edit.
```

Dependencies: the MCP SDK and nothing else. Node ≥22 runs the TypeScript
directly; there is no build step.

### What the tests are for

34 tests, none of which need a network or an API key.

The three error-semantics cases were written before the tool existed. Beyond
those, the suite pins the things whose silent drift would invalidate the
comparison rather than break the build: the exact JSON `tools/list` puts on the
wire, the sha256 of the rubric against the string the Camunda BPMN actually sent,
that the tool policy appears *exactly once* per arm, that v2 lifts it verbatim
with no rewording, and that the result rows carry exactly the columns
`score.mjs` reads.

[`test/harness.test.ts`](test/harness.test.ts) drives the whole agent loop
against a stubbed Anthropic and a stubbed Places. Camunda's connector owned that
loop, so camunda-trial never had to test one; owning it means it can be wrong,
and a bug there would produce a plausible accuracy number with no other symptom.

## Honest caveats

Held constant: rubric text (hash-pinned), golden set, scorer, model, max_tokens,
temperature, user-prompt format, model-call cap.

Not eliminable, and load-bearing on how much the comparison is worth:

1. **Agent scaffolding.** Camunda's AI Agent connector wraps the system prompt
   with framing not visible from outside. This runner sends the rubric raw. Some
   part of any difference could live here.
2. **Output enforcement.** Camunda enforced a response JSON schema; the Messages
   API is prompted for JSON and parsed, with one repair turn allowed. In
   practice this cost nothing — 0 repairs across 455 classifications — but it is
   an asymmetry, and a second forced tool was rejected precisely because it
   would have polluted the tool-discipline measurement.
3. **n is small.** 91 posts, one competitor_mentions, zero feature_gaps. Per-bucket
   figures for the rare classes carry no weight.
4. **Arms B and C are one run each.** Their numbers sit inside or just below arm
   A's observed spread; treat the accuracy column for those rows as suggestive.
   The A≡B agreement is the stronger result, because it is post-for-post
   identity rather than a matching aggregate.

## The friction log

[FRICTION.md](FRICTION.md) is a first-class deliverable, not a postscript. The
MCP TypeScript SDK is on 1.30.0, released ten days before this was built and
outside the assistant's training data. Every entry came from reading the
installed `.d.ts` rather than from recall, including the two that changed the
architecture and the one where a rubric that looked byte-identical was not.

## Licence

Apache-2.0.