# agent-orchestration-mcp-eval

A zero-dependency Model Context Protocol (MCP) server that exposes three deterministic policy tools: seat routing with budget admission, agent-response contract checks, and dossier URL/markdown linting. The same decision code runs behind the MCP server, a native eval harness, and a Promptfoo provider, so the evidence a reviewer sees is the evidence the server enforces.

- **Runtime:** Node.js, ES modules, no `dependencies` in `package.json`. Declared `engines.node >=20`; verified on Node 22.14.0 only.
- **Protocol:** JSON-RPC 2.0 over stdio (newline-delimited) and HTTP (`POST /mcp`). Implements `initialize`, `ping`, `tools/list`, `tools/call`, `resources/list`, `resources/read`. Request IDs must be strings or integers. HTTP refuses unsupported `MCP-Protocol-Version` headers (400) and JSON-RPC batch bodies (400).
- **License:** MIT.

## Verified status

| Check | Command | Result (Node 22.14.0) |
| :--- | :--- | :--- |
| Unit, protocol, stdio and HTTP tests | `npm test` (runs `node --test`) | 38/38 pass |
| Native deterministic eval | `npm run eval` | 64/64 pass, 0 false accepts, 0 false rejects |
| Promptfoo eval on the same fixtures | `npx promptfoo@0.120.19 eval -c promptfooconfig.yaml` | 64/64 pass |
| Sanitization gate | `npm run sanitize` | 0 hits in the tree and in every reachable commit (author, committer, message) |

Not yet verified: connecting a third-party MCP client (Claude Desktop, Cursor) to this server. The configuration below follows each client's documented `mcpServers` format but has not been exercised.

## Architecture

```mermaid
flowchart LR
  Client["MCP client<br/>(Claude Desktop, Cursor, custom)"] -->|"stdio: newline-delimited JSON-RPC"| Stdio["src/server.mjs"]
  Client -->|"HTTP: POST /mcp"| Http["src/http.mjs"]
  Stdio --> Core["src/jsonrpc.mjs<br/>JSON-RPC 2.0 + MCP methods"]
  Http --> Core
  Core --> Registry["src/tools.mjs<br/>registry + input schema validation"]
  Core --> Resources["src/resources.mjs<br/>contracts, fixtures, scorecard"]
  Registry --> Policy["src/policy.mjs<br/>seat routing, reserve floor, circuit breaker"]
  Registry --> Evaluate["src/evaluate.mjs<br/>anchor, schema, claims, citations"]
  Registry --> Dossier["src/dossier.mjs<br/>URL canonicalization, dedup, lint"]
  Native["evals/run-evals.mjs<br/>npm run eval"] --> Registry
  PF["promptfooconfig.yaml"] --> Provider["evals/promptfoo-provider.mjs"] --> Registry
  Fixtures[("evals/cases.json<br/>shared fixtures")] --> Native
  Fixtures --> PF
```

Design rules:

- **Pure core.** `policy.mjs`, `evaluate.mjs` and `dossier.mjs` do no I/O and no clock reads. Same input, same output.
- **Fail closed.** Unknown roles, modes, tiers and malformed accounting values are refused. Accounting values must be finite numbers, checked against the schema before any arithmetic. A negative `usedCU` cannot widen capacity.
- **Schema subset enforced.** Tool input schemas and response contracts use a small JSON Schema subset. Any keyword outside that subset throws at registration instead of being ignored.
- **Transport-free decisions.** The MCP server, native harness and Promptfoo provider all call `callTool()`, so there is one code path to test.

## Tools

Full schemas: `resources/read` on `agent-eval://contracts/tools`.

### `route_agent_seat`

Routes a worker role to a cost tier, then admits its planned spend against a budget snapshot.

| Field | Type | Notes |
| :--- | :--- | :--- |
| `role` | enum | `grunt` (cheap, 0.09 CU), `scout` (mid, 0.9), `mechanic` and `sniper` (heavy, 1.8), `auditor` (judgment, 3.6). |
| `mode` | enum | `read-only` (default) or `workspace-write`. `auditor` is hard-locked to `read-only`. |
| `budget.usedCU` | number ≥ 0 | CU already spent in the window. |
| `budget.capacityCU` | number > 0 | Window capacity in CU. |
| `budget.reservePct` | number 0–100 | Share of capacity held back. agent share = capacity × (100 − reservePct) / 100. |
| `budget.judgmentUsedCU`, `budget.judgmentAllowanceCU` | number ≥ 0 | Judgment-tier spend needs an allowance. Default 0 refuses judgment seats. |
| `budget.consecutiveFailures`, `budget.failureThreshold` | integer | The circuit breaker opens at `failureThreshold` (default 3) and refuses every admission. |

`cu` and `tier` come from the route, never from the caller. Unknown budget keys are refused.

### `evaluate_agent_response`

Checks one response against a contract. Each check reports `pass`, `fail`, or `skipped` (not configured).

| Check | Contract field | Rule |
| :--- | :--- | :--- |
| Anchor | `anchor` (required) | First non-empty line must be exactly `ANCHOR: <anchor>`. |
| Schema | `schema` | The first ```` ```json ```` block must parse and satisfy the schema subset. |
| Forbidden claims | `forbiddenClaims` | Case-insensitive literal phrases. Any hit fails. |
| Coordinate slices | `source` (array of lines) | Each `slice: L<a>-L<b> "<text>"` line must equal source lines a..b joined by newlines. Out-of-range or mismatched citations fail. |

### `dedup_and_lint_dossier`

Accepts `urls`, `markdown`, or both. `ok` is false on any duplicate, invalid URL, or lint violation.

- **URL canonicalization:** lowercase scheme and host, default ports removed, fragment removed, `utm_*`, `fbclid`, `gclid`, `mc_cid`, `mc_eid`, `ref`, `ref_src` removed, query keys sorted, trailing slash removed from non-root paths. Hosts are not rewritten (no `www.` stripping).
- **Dossier lint:** H1 title first; `## Summary`, `## Findings`, `## Sources` in that order; at least one source link; every link canonical and unique within Sources; no bare URLs outside link syntax; no `TODO`, `TBD`, `FIXME` or lorem placeholders.

## Quickstart

```bash
git clone <this-repo> agent-orchestration-mcp-eval
cd agent-orchestration-mcp-eval
npm test              # unit, protocol, stdio and HTTP tests, then the native eval, then the sanitization gate
npm run eval          # native eval only; writes evals/results/scorecard.{md,json}
npm start             # MCP over stdio
npm run start:http    # MCP over HTTP on 127.0.0.1:3333 (override with PORT)
```

Promptfoo:

```bash
npx promptfoo@0.120.19 eval -c promptfooconfig.yaml
```

The version is pinned because the latest Promptfoo release requires Node 22.22 or newer. The config needs no API keys: the provider is a local deterministic function over the same `evals/cases.json`.

## Evaluation results

Measured by `npm run eval` on fixtures version 2 (`evals/results/scorecard.md`). These fixtures are authored in this repo to pin known behaviour. They are a regression suite, not an external benchmark, and 100% here says the code matches the fixtures, not that it generalises.

| Category | Pass | Should-accept | Should-reject | False accepts | False rejects |
| :--- | ---: | ---: | ---: | ---: | ---: |
| Seat routing | 10/10 | 5 | 5 | 0 | 0 |
| Budget admission (reserve, judgment, breaker, accounting bypasses) | 19/19 | 5 | 14 | 0 | 0 |
| Input schema compliance | 4/4 | 0 | 4 | 0 | 0 |
| Evidence acceptance | 4/4 | 4 | 0 | 0 | 0 |
| Hallucination rejection | 13/13 | 0 | 13 | 0 | 0 |
| Dossier dedup and lint | 14/14 | 3 | 11 | 0 | 0 |
| **Total** | **64/64** | **17** | **47** | **0** | **0** |

## MCP client configuration

Replace `<ABSOLUTE_PATH_TO_REPO>` with the absolute path to this repository.

Claude Desktop (`%APPDATA%\Claude\claude_desktop_config.json` on Windows):

```json
{
  "mcpServers": {
    "agent-eval-policy": {
      "command": "node",
      "args": ["<ABSOLUTE_PATH_TO_REPO>/src/server.mjs"]
    }
  }
}
```

Cursor (`.cursor/mcp.json` in a project, or `~/.cursor/mcp.json` globally):

```json
{
  "mcpServers": {
    "agent-eval-policy": {
      "command": "node",
      "args": ["<ABSOLUTE_PATH_TO_REPO>/src/server.mjs"]
    }
  }
}
```

## Limits

- Compute-unit weights (0.09 / 0.9 / 1.8 / 3.6) are relative planning weights. This server does not read provider quota or billing.
- Admission is a function of the caller's snapshot. The server keeps no ledger, so concurrent callers must reconcile usage themselves.
- The HTTP transport has no authentication. It binds to `127.0.0.1` and rejects non-localhost browser origins, but it is not meant to be exposed beyond the local machine.
- No server-initiated requests, no SSE streaming, no session management. Requests are answered in arrival order over stdio.
- Batches are accepted over stdio but refused over HTTP. Batching was removed in MCP 2025-06-18, and the HTTP transport does not implement it.
- The sanitization gate's private-term list is a file outside the repo, passed with `SANITIZE_PRIVATE_TERMS`. Without it the gate runs only the generic rules (emails, phone numbers, keys, local user paths).
- Resources read files from `evals/`. `agent-eval://scorecard/latest` returns nothing until `npm run eval` has run.