MCP server authoring

The three fleet tools from loop-engineering (count, rank by one stored field, fetch one device, over a synthetic fleet of 400 connected medical devices) moved out of the agent's process and into a Model Context Protocol server, so the same agent, task and model can be run with the tools handed over directly or reached over a protocol boundary, and the two compared: what the model is shown, whether it behaves differently, what a call costs in time, and what failure looks like from each side. The writeup lives on shamirai.ai.

Run it

npm install
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env   # gitignored; only the agent needs it, the server never sees it

npm run smoke -- --effort low                # one direct run, no MCP, to prove the rig
npm run experiment -- --condition mcp --effort low --n 5
npm run report                               # the results table below, from runs/*.jsonl
npm run spend                                # total API spend so far
npm run probe                                # the no-model measurements (free), writes runs/probe.json

The server on its own:

npm run server        # McpServer (zod schemas), stdio; waits for JSON-RPC on stdin, logs to stderr
npm run server:raw    # low-level Server, handed the exact JSON Schema from src/rig/tools.mjs

It speaks newline-delimited JSON-RPC on stdin and stdout, so it can be driven by hand:

printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"by-hand","version":"0"}}}' \
  '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"count_devices","arguments":{"max_hours":500}}}' \
  | node src/server/fleet-server.mjs

Any MCP client that can launch a stdio server can use it with the command node src/server/fleet-server.mjs. Nothing here registers it with any client.

Related MCP server: FHIR MCP Server

Layout

src/rig/          shared agent rig: fleet and tools, task and ground truth, hand-written loop, grader
src/server/       fleet-server.mjs (McpServer + zod), fleet-server-raw.mjs (low-level Server), faults.mjs
src/mcp/client.mjs  spawns a server, converts its tools to Anthropic definitions, maps errors, optional restart
scripts/          experiment.mjs (model runs), probe.mjs (no-model measurements), report.mjs, spend.mjs
runs/             every run record as JSONL, plus probe.json

Experiment design

Held fixed across every condition: the fleet, the three tool handlers, the 10-row cap, the task (among devices under 500 operating hours, which had the most unplanned downtime per 100 hours, and how does its vendor risk score compare with the highest-risk device of the same model; correct answer AV3-007 vs AV3-024), the system prompt, and the hand-written loop in src/rig/agent.mjs (prompt caching on, parallel tool calls allowed, 40-turn cap). Only the path between the loop and the handlers changes.

The client (src/mcp/client.mjs) turns each MCP tool into {name, description, input_schema}, dropping MCP-only fields, and maps both failure channels onto the rig's {content, isError}: a result with isError: true, and a callTool that throws (timeout, closed connection, JSON-RPC error), whose message is passed to the model as an error result.

Primary model claude-opus-5 at low and high effort. A secondary arm on claude-haiku-4-5 (no thinking) repeats direct vs mcp, as a less capable reader of the same definitions. Every run is graded by src/rig/grade.mjs from its FINAL: line. Runs were strictly sequential.

scripts/probe.mjs measures what needs no model: the tools/list payload as sent on the wire and as parsed by the client, a field-level diff against toolDefs, definition token counts from the token counting endpoint, per-call latency over 270 warm calls, server startup, the same eight malformed calls sent down all three paths, and five faults (slow tool, crash with and without restart, a newline-terminated log on stdout, an unterminated write on stdout).

Not covered: the Messages API's remote MCP connector, which needs a publicly reachable server URL (this server is stdio only and is never exposed); Streamable HTTP transport; more than one task. Per-call times in run records use a millisecond clock, so the probe's figures are the precise ones.

Results

Model runs, from npm run report. Means exclude runs that ended in api_error. Per-call times come from a millisecond clock inside real runs; mcp-slow includes the 1.5 s timeout.

Notes on the table:

• Every mcp-crash run answered FINAL: IL7-032 vs IL7-032, the task's known shortcut answer, and every one said in its text that the tools had dropped out. The crash landed on the 6th call in all 7 runs (turn 2 in 6 of them), after which the model made 11 to 21 more calls, all failing.

• In mcp-slow the model reissued the identical timed-out call in the next turn in 5/5 runs.

• mcp-crash-restart: one restart per run; the retried call took 94 to 114 ms.

• Haiku 4.5 gave the same wrong answer on both paths. Its call-count gap comes from turn 2: the direct path fetched all ten ranked devices in 2 of 6 runs, the MCP path in 0 of 6.

• The turn-1 tool calls were identical in all 23 runs of the first Opus matrix, across all three paths.

No-model probe (runs/probe.json):

What changed in the mcp definitions: every integer property gained minimum: -9007199254740991 and maximum: 9007199254740991 (including limit, described as 1 to 10); each input_schema gained "$schema": "http://json-schema.org/draft-07/schema#", sent first on the wire and moved last by the client SDK's parse; description moved ahead of type in every described property; and each tool carried execution: {taskSupport: "forbidden"}, which the converter drops.

The same malformed calls down each path (content shown is what the model would read):

Faults, all on the mcp server:

Spend and what was cut

61 run records, $3.85 total API spend, all runs sequential. The API credit on the shared key ran out mid-way through the last batch (credit balance is too low, HTTP 400), which cut three things: the mcp-crash-explained condition was never run (two attempts before it were launched with the client option unwired, so the model saw the SDK's messages exactly as in mcp-crash; they are recorded as mcp-crash runs with a note), Haiku stopped at 6 complete runs per path instead of 10, and the 7 api_error records from those attempts stay in runs/ and are excluded from the means.