mcp-grounding

The verify_place grounding tool from camunda-trial, re-exposed as an MCP server and measured against the same 91-post golden set, with the same deterministic scorer, so the two protocols can be compared on numbers.

The server is the smaller half. The comparison is the point.

Headline: across three runs of the parity arm, the MCP path scores a mean of 87.9% against the Camunda connector's 86.8% — a 1.1-point gap inside a 2.2-point run-to-run spread. On this task, moving the tool between protocols did not change what the model got right. What it changed was smaller, reproducible, and more interesting than the accuracy number.

The numbers

Same rubric (hash-pinned identical), same golden set, same model (claude-sonnet-5), same scorer — eval/score.mjs, copied byte-identical from camunda-trial and never edited. No model anywhere in the judging loop.

Individual runs: 89.0%, 86.8%, 87.9%. Temperature is unset in both systems, so both are non-deterministic; a single run cannot distinguish a real difference from noise, which is why the parity arm was run three times. One of those runs reproduced Camunda's 79/91 exactly.

Read that as: no detectable difference in accuracy. The gap is smaller than the instrument's own resolution. Anyone reporting a single 89.0% run as "MCP beats Camunda by 2.2 points" would be reporting noise.

The confusion matrices are near-identical too. The 86.8% MCP run differs from the Camunda run by exactly one post traded between buckets.

Related MCP server: MCP Server

The one reproducible difference

MCP calls the tool on a post Camunda never touched, in every single run, across every arm:

1ute6a4 — "Has anyone visited Sedona, Arizona? Worth the trip?" — golden no_signal

Camunda: no tool call, labelled no_signal, correct. MCP: verifies Sedona (found=true) every time, and in 2 of 3 parity runs then labels it rising_destinations — wrong.

Nothing in the prompt differs. Same rubric bytes, same model, same policy text. The difference is where the tool sits: in MCP the tool definition travels in the request payload on every turn, and that appears to make it marginally more reachable. One post out of 91 is not a law of nature, but 5 runs out of 5 is not noise either.

This is the shape of finding the project was built to catch — too small to move the headline number, invisible without a per-call trace, and a real behavioural difference between the two protocols.

What the tool description turned out to be worth

The build brief assumed the tool description would matter a great deal in MCP, because "in the Camunda version, the BPMN wiring decided when the tool fired."

That premise was wrong, and finding out was the useful part. The Camunda rubric already contained a ## Tool policy: verify_place section — the system prompt decided when the tool fired, not the BPMN. The connector's own description was one sentence that Camunda Modeler read and no model ever did.

So the real question is not "does a description matter" but "does the spec survive being moved into one". Three arms, same instruction, only its location changing. Arm C is a negative control where the policy exists nowhere at all.

The last column counts every tool call on a post whose golden label is not rising_destinations. It is deliberately broader than the scorer's own restraint metric, which looks only at golden no_signal posts and so would not have seen arm B verifying Viator on a competitor_mentions post.

Moving the policy changed nothing. On the 31-post day-2 subset, arms A and B agreed on every post — same label, same tool-call count, 31 for 31. At n=91 arm B lands inside arm A's range. The spec survives the move intact.

Removing the policy did not break grounding. It broke restraint. Arm C still grounded every rising_destinations prediction it made. What degraded was the "do not call it otherwise" half: spurious calls doubled.

That split says the two halves of the policy are held up by different things. The obligation to ground has a backstop — the rubric's ## Output section still asks for groundedPlaces "for each place verified with found=true", so the requirement is implied by the output contract even with the policy deleted. Restraint has no backstop, and it is the first thing to go.

The failure the brief predicted never appeared. Not once across five runs and 455 classifications did the model assign rising_destinations without verifying first — including in the arm where no policy existed anywhere. On this rubric the grounding obligation is over-determined.

Every description version and its measurement: docs/TOOL_DESCRIPTIONS.md.

Where the design decisions were

Error semantics: a place that doesn't exist is not an error

The contract carried over from the Java connector, and the reason the tests were written first:

The distinction that matters is the last row. MCP offers two different "errors", and the tempting one is wrong here. A tool result with isError: true is handed back to the model as text — so an expired API key becomes something the model can reason around, and it can still answer rising_destinations having verified nothing. A JSON-RPC error never reaches the model at all; the client's callTool() rejects and the eval records the row as errored, which is exactly where a Camunda incident put it.

The SDK makes the right shape hard to reach. McpServer.registerTool() catches every error a tool callback throws and rewrites it into { isError: true }, discarding error.code and error.data — only the message survives. A protocol-level failure is unreachable from a tool callback. This server is therefore built on the low-level Server class, where the error code survives to the client. See FRICTION.md.

Asserted end-to-end over a real transport in test/error-semantics.test.ts, including the behavioural claim: an infrastructure failure never reaches the model as a tool result.

Which primitive is each thing

Decided by a type signature rather than by taste:

• verify_place is a tool. It performs an action with a side effect — an outbound call spending someone's quota — and the model chooses when.

• The rubric is a resource (grounding://rubric/v3), not a prompt. MCP's PromptMessage role is "user" | "assistant" — there is no system role. The rubric is a system prompt; returning it as a user message would move 13,688 characters of instruction out of the system position and quietly change the experiment. A resource carries text with no role attached and lets the client place it.

• classify_post is a prompt. The hand-driving entry point for Claude Code. It carries the post, not the rubric — and why it cannot carry the rubric is the finding above.

The credential is not a tool input

The Java connector took apiKey as an input property bound to {{secrets.GOOGLE_PLACES_API_KEY}}, so it appeared as a field on the element template. Here it is server configuration read from the environment and is absent from the tool schema — the model can neither see it nor be talked into supplying one. A test asserts no credential appears in the published schema.

What "grounded" is actually worth

Worth stating plainly, because the project exists to avoid shipping plausible verifications nobody checked — and the verification layer has its own false positives.

Google Places Text Search is a search, not an existence check. It matches on substrings and business names:

So found: true means "Places returned a best match for this query string", not "this place exists".

This is not hypothetical. In arm B the model verified Viator — a tour booking platform, not a place — and Google returned found=true.

The tool is left unchanged: tightening it would diverge from the Java connector and break the only thing making the two numbers comparable, and the golden set never exercises the false-positive path. But it bounds the claim. "Perfect tool discipline" means the tool was called, never that the place was real.

eval/live-check.ts reports these cases explicitly rather than hiding them.

Running it

npm install

cp .env.example .env

Fill in GOOGLE_PLACES_API_KEY (Places API New enabled) and ANTHROPIC_API_KEY. .env is gitignored.

npm test

Day-1 check against the live API — confirms a nonexistent place returns a verdict, not an error:

node --env-file=.env eval/live-check.ts

A full 91-post run, then score it with the untouched scorer:

node --env-file=.env eval/run.ts --arm full --description v1-parity --label myrun

node eval/score.mjs eval/results/run-<timestamp>-myrun/results.jsonl

Compare finished runs (this calls score.mjs, it does not compute accuracy itself):

node eval/summarise.mjs

Attaching to Claude Code

.mcp.json is committed and loads the key via --env-file, so no secret enters a tracked file. Claude Code reads it at startup.

How it is put together

src/server/server.ts        MCP server on the low-level Server class
src/server/tool-description.ts  versioned descriptions, append-only
src/server/rubric.ts        both arms, split from one source so they cannot drift
src/places/client.ts        port of PlacesClient.java, same field mask, same codes
eval/run.ts                 MCP client + Messages API; tool def pulled from tools/list
eval/classify.ts            the agent loop, extracted so it can be tested
eval/score.mjs              copied byte-identical from camunda-trial. Never edit.

Dependencies: the MCP SDK and nothing else. Node ≥22 runs the TypeScript directly; there is no build step.

What the tests are for

34 tests, none of which need a network or an API key.

The three error-semantics cases were written before the tool existed. Beyond those, the suite pins the things whose silent drift would invalidate the comparison rather than break the build: the exact JSON tools/list puts on the wire, the sha256 of the rubric against the string the Camunda BPMN actually sent, that the tool policy appears exactly once per arm, that v2 lifts it verbatim with no rewording, and that the result rows carry exactly the columns score.mjs reads.

test/harness.test.ts drives the whole agent loop against a stubbed Anthropic and a stubbed Places. Camunda's connector owned that loop, so camunda-trial never had to test one; owning it means it can be wrong, and a bug there would produce a plausible accuracy number with no other symptom.

Honest caveats

Held constant: rubric text (hash-pinned), golden set, scorer, model, max_tokens, temperature, user-prompt format, model-call cap.

Not eliminable, and load-bearing on how much the comparison is worth:

1. Agent scaffolding. Camunda's AI Agent connector wraps the system prompt with framing not visible from outside. This runner sends the rubric raw. Some part of any difference could live here.

2. Output enforcement. Camunda enforced a response JSON schema; the Messages API is prompted for JSON and parsed, with one repair turn allowed. In practice this cost nothing — 0 repairs across 455 classifications — but it is an asymmetry, and a second forced tool was rejected precisely because it would have polluted the tool-discipline measurement.

3. n is small. 91 posts, one competitor_mentions, zero feature_gaps. Per-bucket figures for the rare classes carry no weight.

4. Arms B and C are one run each. Their numbers sit inside or just below arm A's observed spread; treat the accuracy column for those rows as suggestive. The A≡B agreement is the stronger result, because it is post-for-post identity rather than a matching aggregate.

The friction log

FRICTION.md is a first-class deliverable, not a postscript. The MCP TypeScript SDK is on 1.30.0, released ten days before this was built and outside the assistant's training data. Every entry came from reading the installed .d.ts rather than from recall, including the two that changed the architecture and the one where a rubric that looked byte-identical was not.

Licence

Apache-2.0.