agent-eval-policy
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@agent-eval-policyRoute a sniper to workspace-write with 4.5 of 10 CU used, 10% reserve"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
agent-orchestration-mcp-eval
A zero-dependency Model Context Protocol (MCP) server that exposes three deterministic policy tools: seat routing with budget admission, agent-response contract checks, and dossier URL/markdown linting. The same decision code runs behind the MCP server, a native eval harness, and a Promptfoo provider, so the evidence a reviewer sees is the evidence the server enforces.
Runtime: Node.js, ES modules, no
dependenciesinpackage.json. Declaredengines.node >=20; verified on Node 22.14.0 only.Protocol: JSON-RPC 2.0 over stdio (newline-delimited) and HTTP (
POST /mcp). Implementsinitialize,ping,tools/list,tools/call,resources/list,resources/read. Request IDs must be strings or integers. HTTP refuses unsupportedMCP-Protocol-Versionheaders (400) and JSON-RPC batch bodies (400).License: MIT.
Verified status
Check | Command | Result (Node 22.14.0) |
Unit, protocol, stdio and HTTP tests |
| 38/38 pass |
Native deterministic eval |
| 64/64 pass, 0 false accepts, 0 false rejects |
Promptfoo eval on the same fixtures |
| 64/64 pass |
Sanitization gate |
| 0 hits in the tree and in every reachable commit (author, committer, message) |
Not yet verified: connecting a third-party MCP client (Claude Desktop, Cursor) to this server. The configuration below follows each client's documented mcpServers format but has not been exercised.
Related MCP server: APIVouch
Architecture
flowchart LR
Client["MCP client<br/>(Claude Desktop, Cursor, custom)"] -->|"stdio: newline-delimited JSON-RPC"| Stdio["src/server.mjs"]
Client -->|"HTTP: POST /mcp"| Http["src/http.mjs"]
Stdio --> Core["src/jsonrpc.mjs<br/>JSON-RPC 2.0 + MCP methods"]
Http --> Core
Core --> Registry["src/tools.mjs<br/>registry + input schema validation"]
Core --> Resources["src/resources.mjs<br/>contracts, fixtures, scorecard"]
Registry --> Policy["src/policy.mjs<br/>seat routing, reserve floor, circuit breaker"]
Registry --> Evaluate["src/evaluate.mjs<br/>anchor, schema, claims, citations"]
Registry --> Dossier["src/dossier.mjs<br/>URL canonicalization, dedup, lint"]
Native["evals/run-evals.mjs<br/>npm run eval"] --> Registry
PF["promptfooconfig.yaml"] --> Provider["evals/promptfoo-provider.mjs"] --> Registry
Fixtures[("evals/cases.json<br/>shared fixtures")] --> Native
Fixtures --> PFDesign rules:
Pure core.
policy.mjs,evaluate.mjsanddossier.mjsdo no I/O and no clock reads. Same input, same output.Fail closed. Unknown roles, modes, tiers and malformed accounting values are refused. Accounting values must be finite numbers, checked against the schema before any arithmetic. A negative
usedCUcannot widen capacity.Schema subset enforced. Tool input schemas and response contracts use a small JSON Schema subset. Any keyword outside that subset throws at registration instead of being ignored.
Transport-free decisions. The MCP server, native harness and Promptfoo provider all call
callTool(), so there is one code path to test.
Tools
Full schemas: resources/read on agent-eval://contracts/tools.
route_agent_seat
Routes a worker role to a cost tier, then admits its planned spend against a budget snapshot.
Field | Type | Notes |
| enum |
|
| enum |
|
| number ≥ 0 | CU already spent in the window. |
| number > 0 | Window capacity in CU. |
| number 0–100 | Share of capacity held back. agent share = capacity × (100 − reservePct) / 100. |
| number ≥ 0 | Judgment-tier spend needs an allowance. Default 0 refuses judgment seats. |
| integer | The circuit breaker opens at |
cu and tier come from the route, never from the caller. Unknown budget keys are refused.
evaluate_agent_response
Checks one response against a contract. Each check reports pass, fail, or skipped (not configured).
Check | Contract field | Rule |
Anchor |
| First non-empty line must be exactly |
Schema |
| The first |
Forbidden claims |
| Case-insensitive literal phrases. Any hit fails. |
Coordinate slices |
| Each |
dedup_and_lint_dossier
Accepts urls, markdown, or both. ok is false on any duplicate, invalid URL, or lint violation.
URL canonicalization: lowercase scheme and host, default ports removed, fragment removed,
utm_*,fbclid,gclid,mc_cid,mc_eid,ref,ref_srcremoved, query keys sorted, trailing slash removed from non-root paths. Hosts are not rewritten (nowww.stripping).Dossier lint: H1 title first;
## Summary,## Findings,## Sourcesin that order; at least one source link; every link canonical and unique within Sources; no bare URLs outside link syntax; noTODO,TBD,FIXMEor lorem placeholders.
Quickstart
git clone <this-repo> agent-orchestration-mcp-eval
cd agent-orchestration-mcp-eval
npm test # unit, protocol, stdio and HTTP tests, then the native eval, then the sanitization gate
npm run eval # native eval only; writes evals/results/scorecard.{md,json}
npm start # MCP over stdio
npm run start:http # MCP over HTTP on 127.0.0.1:3333 (override with PORT)Promptfoo:
npx promptfoo@0.120.19 eval -c promptfooconfig.yamlThe version is pinned because the latest Promptfoo release requires Node 22.22 or newer. The config needs no API keys: the provider is a local deterministic function over the same evals/cases.json.
Evaluation results
Measured by npm run eval on fixtures version 2 (evals/results/scorecard.md). These fixtures are authored in this repo to pin known behaviour. They are a regression suite, not an external benchmark, and 100% here says the code matches the fixtures, not that it generalises.
Category | Pass | Should-accept | Should-reject | False accepts | False rejects |
Seat routing | 10/10 | 5 | 5 | 0 | 0 |
Budget admission (reserve, judgment, breaker, accounting bypasses) | 19/19 | 5 | 14 | 0 | 0 |
Input schema compliance | 4/4 | 0 | 4 | 0 | 0 |
Evidence acceptance | 4/4 | 4 | 0 | 0 | 0 |
Hallucination rejection | 13/13 | 0 | 13 | 0 | 0 |
Dossier dedup and lint | 14/14 | 3 | 11 | 0 | 0 |
Total | 64/64 | 17 | 47 | 0 | 0 |
MCP client configuration
Replace <ABSOLUTE_PATH_TO_REPO> with the absolute path to this repository.
Claude Desktop (%APPDATA%\Claude\claude_desktop_config.json on Windows):
{
"mcpServers": {
"agent-eval-policy": {
"command": "node",
"args": ["<ABSOLUTE_PATH_TO_REPO>/src/server.mjs"]
}
}
}Cursor (.cursor/mcp.json in a project, or ~/.cursor/mcp.json globally):
{
"mcpServers": {
"agent-eval-policy": {
"command": "node",
"args": ["<ABSOLUTE_PATH_TO_REPO>/src/server.mjs"]
}
}
}Limits
Compute-unit weights (0.09 / 0.9 / 1.8 / 3.6) are relative planning weights. This server does not read provider quota or billing.
Admission is a function of the caller's snapshot. The server keeps no ledger, so concurrent callers must reconcile usage themselves.
The HTTP transport has no authentication. It binds to
127.0.0.1and rejects non-localhost browser origins, but it is not meant to be exposed beyond the local machine.No server-initiated requests, no SSE streaming, no session management. Requests are answered in arrival order over stdio.
Batches are accepted over stdio but refused over HTTP. Batching was removed in MCP 2025-06-18, and the HTTP transport does not implement it.
The sanitization gate's private-term list is a file outside the repo, passed with
SANITIZE_PRIVATE_TERMS. Without it the gate runs only the generic rules (emails, phone numbers, keys, local user paths).Resources read files from
evals/.agent-eval://scorecard/latestreturns nothing untilnpm run evalhas run.
Available Tools
3 toolsdedup_and_lint_dossierA
Canonicalize and deduplicate URLs, and/or lint a markdown dossier against the dossier schema. ok is false on any duplicate, invalid URL, or lint violation.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | No | URLs to canonicalize and deduplicate. | |
| markdown | No | Dossier markdown to lint. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full load. It does disclose one genuinely useful behavioral trait beyond structured data: failures surface via an 'ok' flag rather than an error, triggered by duplicates, invalid URLs, or lint violations. However, it never says whether the tool mutates any stored dossier or is purely a read/compute operation, nor whether duplicate detection is order-sensitive or per-host.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with the two capabilities front-loaded and the failure condition last; no filler. The 'and/or' phrasing is slightly muddy but compactness is good.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a small two-optional-parameter tool with no output schema and no annotations, the description explains the failure signal but leaves gaps: it never says what a successful response contains (canonicalized URL list? lint finding details?) or what the referenced 'dossier schema' is. That reference is the most important unexplained term in the definition.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters, and the description only restates that urls are canonicalized/deduplicated and markdown is linted. It adds no format details, such as expected URL scheme or how the markdown is validated against the 'dossier schema'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names specific verbs (canonicalize, deduplicate, lint) against specific resources (URLs, a markdown dossier), and the 'and/or' makes clear it is two modes in one tool. The sibling tools (route_agent_seat, evaluate_agent_response) are unrelated operations, so no differentiation is needed.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied by the parameter split: pass 'urls' to dedupe, 'markdown' to lint, or both. There is no explicit statement of when to call this versus doing the work inline, no mention of prerequisites, and no guidance on what happens if neither parameter is supplied (both are optional).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
evaluate_agent_responseB
Check one agent response against a contract: anchor line, JSON payload schema, forbidden claims, and verified source-slice citations.
| Name | Required | Description | Default |
|---|---|---|---|
| contract | Yes | ||
| response | Yes | Full text of the agent response. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations supplied, the description carries the full behavioral burden. It says what is checked but not what happens on a violation (throw, return a verdict object, list of failures), whether checks short-circuit, or whether 'verified source-slice citations' means the tool fetches/validates external sources — a potentially expensive or failing operation left undisclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with a colon-delimited enumeration — efficient and scannable, with no filler. Density borders on terseness; the facet labels are compressed enough that a reader must infer what 'verified' and 'anchor line' mean operationally.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has two top-level params but a four-field nested contract, no annotations, and no output schema, so the description must cover inputs and outputs alone. Inputs are reasonably covered; the return value — the entire point of an evaluation/check tool — is completely unspecified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 50%: only 'response' is documented in the schema, while none of the contract's nested properties (anchor, schema, source, forbiddenClaims) are. The description partially compensates by mapping the four facets to those fields and hinting that the anchor is a line and that source slices are verified, but it gives no format details for anchor, schema, or forbiddenClaims entries.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Check one agent response against a contract') and enumerates the four contract facets it evaluates: anchor line, JSON payload schema, forbidden claims, and source-slice citations. The scope word 'one' signals single-response evaluation, distinguishing it from batch-oriented siblings like dedup_and_lint_dossier, though no sibling is named explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: 'Check one agent response against a contract' suggests a per-response validation call, likely after an agent has produced output. There is no explicit statement of when to prefer this over dedup_and_lint_dossier, no preconditions, and no note on whether the response must already exist or be retrievable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
route_agent_seatB
Route a worker role to a cost tier and admit its planned spend against the budget snapshot. Fails closed on unknown roles, modes, or malformed accounting values.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | read-only (default) or workspace-write. | |
| role | Yes | Worker role. auditor is always read-only. | |
| budget | Yes | Accounting snapshot for the admission decision. All values are supplied by the caller and must be finite. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations supplied, the description carries the full burden, and it does disclose one genuinely useful trait: it fails closed on unknown roles, modes, or malformed accounting values. However, it never says whether the call is purely a check or whether it persists/consumes budget, nor whether it needs any authorization, nor how the circuit-breaker and judgment-allowance inputs affect the decision. For an unannotated tool with a nested accounting object, that leaves meaningful behavioral gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, the primary action front-loaded and the error contract second. There is no filler, no restatement of the tool name, and nothing that could be cut without losing information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool takes a nested seven-field accounting object plus two enums and has no output schema, so the description should clarify what an admission decision yields (tier granted, allowance, or refusal) and how the breaker fields interact. It covers the failure contract but not the success contract, leaving the agent unable to predict the response shape. Adequate but with a clear gap given the complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, including enum explanations for role and mode and per-field constraints on the budget object, so the schema already does the heavy lifting and baseline 3 applies. The description adds only loose framing ('budget snapshot', 'unknown roles, modes, malformed accounting values') and no interpretation of the judgment/circuit-breaker fields beyond what the schema states.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action on specific resources: routing a worker role to a cost tier and admitting planned spend against a budget snapshot. The domain is abstract (seats, tiers, CU), but a reader can tell this is an admissibility/pricing decision rather than a lint or evaluation call, which distinguishes it from the two siblings. It stops short of five because 'route' and 'admit' are internal jargon that a fresh agent may not map to a concrete outcome.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no explicit when-to-use guidance, no statement of prerequisites (for example that a budget snapshot must be assembled by the caller), and no mention of when not to call it. The only usage-adjacent signal is the failure behavior, which tells the agent what happens on bad input, not when this tool is the right choice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
3 tool updates
v0.2.0- First observed
dedup_and_lint_dossier - First observed
evaluate_agent_response - First observed
route_agent_seat
TDQS
Scored across 3 tools
Each tool targets a distinct phase: seat/budget routing, single-response contract checking, and URL/dossier hygiene. The only wrinkle is dedup_and_lint_dossier bundling two operations with 'and/or' semantics, which could make an agent unsure whether it must supply both inputs.
All three follow a clean verb_noun (or verb_noun_noun) snake_case pattern: route_agent_seat, evaluate_agent_response, dedup_and_lint_dossier. Predictable and readable throughout.
Three tools is thin but matches a narrow policy/eval-enforcement role where each tool is a distinct gate. Nothing feels redundant or padded.
Covers routing/admission, response validation, and dossier hygiene, but there is no way to inspect or refresh the budget snapshot, enumerate valid roles/contracts, or manage schema definitions—agents hitting an 'unknown role' or 'fails closed' path have no recovery tool.
Maintenance
Related MCP Connectors
Pre-execution governance for AI agents. Deterministic PASS/FAIL/REVIEW verdicts, replayable proof.
Agent routing, safety preflight, execution receipts, best execution and verified outcomes.
AI agent infrastructure: dedup, cost prediction, validation, governance, failure intelligence.
Tenant-scoped control layer for agent-to-agent systems: governed routing, approvals, evidence.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to run evidence QC gates mid-workflow—checking claims, sources, prose, and vault hygiene deterministically before research ships.MIT
- AlicenseNot gradedqualityBmaintenanceEnables agents to route goals across multiple independent APIs, enforce constraints, reject invalid evidence, and return verified outcomes with tamper-evident receipts.MIT
- AlicenseAqualityBmaintenanceEnables autonomous AI agents to proactively poll event queues, evaluate document compliance, retrieve grounded knowledge via semantic RAG, enforce human-in-the-loop approval gates, and track multi-model cost budgets.10MIT
- AlicenseCqualityAmaintenancePersistent multi-agent work graph and document-state machine for contracts, specs, slices, evidence, verification, handoffs, and cost-aware AI execution.7Apache 2.0