Skip to main content
Glama
agent-eval-policy

agent-eval-policy

by romanfjose

agent-orchestration-mcp-eval

A zero-dependency Model Context Protocol (MCP) server that exposes three deterministic policy tools: seat routing with budget admission, agent-response contract checks, and dossier URL/markdown linting. The same decision code runs behind the MCP server, a native eval harness, and a Promptfoo provider, so the evidence a reviewer sees is the evidence the server enforces.

  • Runtime: Node.js, ES modules, no dependencies in package.json. Declared engines.node >=20; verified on Node 22.14.0 only.

  • Protocol: JSON-RPC 2.0 over stdio (newline-delimited) and HTTP (POST /mcp). Implements initialize, ping, tools/list, tools/call, resources/list, resources/read. Request IDs must be strings or integers. HTTP refuses unsupported MCP-Protocol-Version headers (400) and JSON-RPC batch bodies (400).

  • License: MIT.

Verified status

Check

Command

Result (Node 22.14.0)

Unit, protocol, stdio and HTTP tests

npm test (runs node --test)

38/38 pass

Native deterministic eval

npm run eval

64/64 pass, 0 false accepts, 0 false rejects

Promptfoo eval on the same fixtures

npx promptfoo@0.120.19 eval -c promptfooconfig.yaml

64/64 pass

Sanitization gate

npm run sanitize

0 hits in the tree and in every reachable commit (author, committer, message)

Not yet verified: connecting a third-party MCP client (Claude Desktop, Cursor) to this server. The configuration below follows each client's documented mcpServers format but has not been exercised.

Related MCP server: APIVouch

Architecture

flowchart LR
  Client["MCP client<br/>(Claude Desktop, Cursor, custom)"] -->|"stdio: newline-delimited JSON-RPC"| Stdio["src/server.mjs"]
  Client -->|"HTTP: POST /mcp"| Http["src/http.mjs"]
  Stdio --> Core["src/jsonrpc.mjs<br/>JSON-RPC 2.0 + MCP methods"]
  Http --> Core
  Core --> Registry["src/tools.mjs<br/>registry + input schema validation"]
  Core --> Resources["src/resources.mjs<br/>contracts, fixtures, scorecard"]
  Registry --> Policy["src/policy.mjs<br/>seat routing, reserve floor, circuit breaker"]
  Registry --> Evaluate["src/evaluate.mjs<br/>anchor, schema, claims, citations"]
  Registry --> Dossier["src/dossier.mjs<br/>URL canonicalization, dedup, lint"]
  Native["evals/run-evals.mjs<br/>npm run eval"] --> Registry
  PF["promptfooconfig.yaml"] --> Provider["evals/promptfoo-provider.mjs"] --> Registry
  Fixtures[("evals/cases.json<br/>shared fixtures")] --> Native
  Fixtures --> PF

Design rules:

  • Pure core. policy.mjs, evaluate.mjs and dossier.mjs do no I/O and no clock reads. Same input, same output.

  • Fail closed. Unknown roles, modes, tiers and malformed accounting values are refused. Accounting values must be finite numbers, checked against the schema before any arithmetic. A negative usedCU cannot widen capacity.

  • Schema subset enforced. Tool input schemas and response contracts use a small JSON Schema subset. Any keyword outside that subset throws at registration instead of being ignored.

  • Transport-free decisions. The MCP server, native harness and Promptfoo provider all call callTool(), so there is one code path to test.

Tools

Full schemas: resources/read on agent-eval://contracts/tools.

route_agent_seat

Routes a worker role to a cost tier, then admits its planned spend against a budget snapshot.

Field

Type

Notes

role

enum

grunt (cheap, 0.09 CU), scout (mid, 0.9), mechanic and sniper (heavy, 1.8), auditor (judgment, 3.6).

mode

enum

read-only (default) or workspace-write. auditor is hard-locked to read-only.

budget.usedCU

number ≥ 0

CU already spent in the window.

budget.capacityCU

number > 0

Window capacity in CU.

budget.reservePct

number 0–100

Share of capacity held back. agent share = capacity × (100 − reservePct) / 100.

budget.judgmentUsedCU, budget.judgmentAllowanceCU

number ≥ 0

Judgment-tier spend needs an allowance. Default 0 refuses judgment seats.

budget.consecutiveFailures, budget.failureThreshold

integer

The circuit breaker opens at failureThreshold (default 3) and refuses every admission.

cu and tier come from the route, never from the caller. Unknown budget keys are refused.

evaluate_agent_response

Checks one response against a contract. Each check reports pass, fail, or skipped (not configured).

Check

Contract field

Rule

Anchor

anchor (required)

First non-empty line must be exactly ANCHOR: <anchor>.

Schema

schema

The first ```json block must parse and satisfy the schema subset.

Forbidden claims

forbiddenClaims

Case-insensitive literal phrases. Any hit fails.

Coordinate slices

source (array of lines)

Each slice: L<a>-L<b> "<text>" line must equal source lines a..b joined by newlines. Out-of-range or mismatched citations fail.

dedup_and_lint_dossier

Accepts urls, markdown, or both. ok is false on any duplicate, invalid URL, or lint violation.

  • URL canonicalization: lowercase scheme and host, default ports removed, fragment removed, utm_*, fbclid, gclid, mc_cid, mc_eid, ref, ref_src removed, query keys sorted, trailing slash removed from non-root paths. Hosts are not rewritten (no www. stripping).

  • Dossier lint: H1 title first; ## Summary, ## Findings, ## Sources in that order; at least one source link; every link canonical and unique within Sources; no bare URLs outside link syntax; no TODO, TBD, FIXME or lorem placeholders.

Quickstart

git clone <this-repo> agent-orchestration-mcp-eval
cd agent-orchestration-mcp-eval
npm test              # unit, protocol, stdio and HTTP tests, then the native eval, then the sanitization gate
npm run eval          # native eval only; writes evals/results/scorecard.{md,json}
npm start             # MCP over stdio
npm run start:http    # MCP over HTTP on 127.0.0.1:3333 (override with PORT)

Promptfoo:

npx promptfoo@0.120.19 eval -c promptfooconfig.yaml

The version is pinned because the latest Promptfoo release requires Node 22.22 or newer. The config needs no API keys: the provider is a local deterministic function over the same evals/cases.json.

Evaluation results

Measured by npm run eval on fixtures version 2 (evals/results/scorecard.md). These fixtures are authored in this repo to pin known behaviour. They are a regression suite, not an external benchmark, and 100% here says the code matches the fixtures, not that it generalises.

Category

Pass

Should-accept

Should-reject

False accepts

False rejects

Seat routing

10/10

5

5

0

0

Budget admission (reserve, judgment, breaker, accounting bypasses)

19/19

5

14

0

0

Input schema compliance

4/4

0

4

0

0

Evidence acceptance

4/4

4

0

0

0

Hallucination rejection

13/13

0

13

0

0

Dossier dedup and lint

14/14

3

11

0

0

Total

64/64

17

47

0

0

MCP client configuration

Replace <ABSOLUTE_PATH_TO_REPO> with the absolute path to this repository.

Claude Desktop (%APPDATA%\Claude\claude_desktop_config.json on Windows):

{
  "mcpServers": {
    "agent-eval-policy": {
      "command": "node",
      "args": ["<ABSOLUTE_PATH_TO_REPO>/src/server.mjs"]
    }
  }
}

Cursor (.cursor/mcp.json in a project, or ~/.cursor/mcp.json globally):

{
  "mcpServers": {
    "agent-eval-policy": {
      "command": "node",
      "args": ["<ABSOLUTE_PATH_TO_REPO>/src/server.mjs"]
    }
  }
}

Limits

  • Compute-unit weights (0.09 / 0.9 / 1.8 / 3.6) are relative planning weights. This server does not read provider quota or billing.

  • Admission is a function of the caller's snapshot. The server keeps no ledger, so concurrent callers must reconcile usage themselves.

  • The HTTP transport has no authentication. It binds to 127.0.0.1 and rejects non-localhost browser origins, but it is not meant to be exposed beyond the local machine.

  • No server-initiated requests, no SSE streaming, no session management. Requests are answered in arrival order over stdio.

  • Batches are accepted over stdio but refused over HTTP. Batching was removed in MCP 2025-06-18, and the HTTP transport does not implement it.

  • The sanitization gate's private-term list is a file outside the repo, passed with SANITIZE_PRIVATE_TERMS. Without it the gate runs only the generic rules (emails, phone numbers, keys, local user paths).

  • Resources read files from evals/. agent-eval://scorecard/latest returns nothing until npm run eval has run.

Available Tools

3 tools
dedup_and_lint_dossierA

Canonicalize and deduplicate URLs, and/or lint a markdown dossier against the dossier schema. ok is false on any duplicate, invalid URL, or lint violation.

NameRequiredDescriptionDefault
urlsNoURLs to canonicalize and deduplicate.
markdownNoDossier markdown to lint.

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full load. It does disclose one genuinely useful behavioral trait beyond structured data: failures surface via an 'ok' flag rather than an error, triggered by duplicates, invalid URLs, or lint violations. However, it never says whether the tool mutates any stored dossier or is purely a read/compute operation, nor whether duplicate detection is order-sensitive or per-host.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with the two capabilities front-loaded and the failure condition last; no filler. The 'and/or' phrasing is slightly muddy but compactness is good.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a small two-optional-parameter tool with no output schema and no annotations, the description explains the failure signal but leaves gaps: it never says what a successful response contains (canonicalized URL list? lint finding details?) or what the referenced 'dossier schema' is. That reference is the most important unexplained term in the definition.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents both parameters, and the description only restates that urls are canonicalized/deduplicated and markdown is linted. It adds no format details, such as expected URL scheme or how the markdown is validated against the 'dossier schema'.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names specific verbs (canonicalize, deduplicate, lint) against specific resources (URLs, a markdown dossier), and the 'and/or' makes clear it is two modes in one tool. The sibling tools (route_agent_seat, evaluate_agent_response) are unrelated operations, so no differentiation is needed.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied by the parameter split: pass 'urls' to dedupe, 'markdown' to lint, or both. There is no explicit statement of when to call this versus doing the work inline, no mention of prerequisites, and no guidance on what happens if neither parameter is supplied (both are optional).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_agent_responseB

Check one agent response against a contract: anchor line, JSON payload schema, forbidden claims, and verified source-slice citations.

NameRequiredDescriptionDefault
contractYes
responseYesFull text of the agent response.

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations supplied, the description carries the full behavioral burden. It says what is checked but not what happens on a violation (throw, return a verdict object, list of failures), whether checks short-circuit, or whether 'verified source-slice citations' means the tool fetches/validates external sources — a potentially expensive or failing operation left undisclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with a colon-delimited enumeration — efficient and scannable, with no filler. Density borders on terseness; the facet labels are compressed enough that a reader must infer what 'verified' and 'anchor line' mean operationally.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has two top-level params but a four-field nested contract, no annotations, and no output schema, so the description must cover inputs and outputs alone. Inputs are reasonably covered; the return value — the entire point of an evaluation/check tool — is completely unspecified.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 50%: only 'response' is documented in the schema, while none of the contract's nested properties (anchor, schema, source, forbiddenClaims) are. The description partially compensates by mapping the four facets to those fields and hinting that the anchor is a line and that source slices are verified, but it gives no format details for anchor, schema, or forbiddenClaims entries.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Check one agent response against a contract') and enumerates the four contract facets it evaluates: anchor line, JSON payload schema, forbidden claims, and source-slice citations. The scope word 'one' signals single-response evaluation, distinguishing it from batch-oriented siblings like dedup_and_lint_dossier, though no sibling is named explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied rather than stated: 'Check one agent response against a contract' suggests a per-response validation call, likely after an agent has produced output. There is no explicit statement of when to prefer this over dedup_and_lint_dossier, no preconditions, and no note on whether the response must already exist or be retrievable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

route_agent_seatB

Route a worker role to a cost tier and admit its planned spend against the budget snapshot. Fails closed on unknown roles, modes, or malformed accounting values.

NameRequiredDescriptionDefault
modeNoread-only (default) or workspace-write.
roleYesWorker role. auditor is always read-only.
budgetYesAccounting snapshot for the admission decision. All values are supplied by the caller and must be finite.

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations supplied, the description carries the full burden, and it does disclose one genuinely useful trait: it fails closed on unknown roles, modes, or malformed accounting values. However, it never says whether the call is purely a check or whether it persists/consumes budget, nor whether it needs any authorization, nor how the circuit-breaker and judgment-allowance inputs affect the decision. For an unannotated tool with a nested accounting object, that leaves meaningful behavioral gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, the primary action front-loaded and the error contract second. There is no filler, no restatement of the tool name, and nothing that could be cut without losing information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool takes a nested seven-field accounting object plus two enums and has no output schema, so the description should clarify what an admission decision yields (tier granted, allowance, or refusal) and how the breaker fields interact. It covers the failure contract but not the success contract, leaving the agent unable to predict the response shape. Adequate but with a clear gap given the complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, including enum explanations for role and mode and per-field constraints on the budget object, so the schema already does the heavy lifting and baseline 3 applies. The description adds only loose framing ('budget snapshot', 'unknown roles, modes, malformed accounting values') and no interpretation of the judgment/circuit-breaker fields beyond what the schema states.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action on specific resources: routing a worker role to a cost tier and admitting planned spend against a budget snapshot. The domain is abstract (seats, tiers, CU), but a reader can tell this is an admissibility/pricing decision rather than a lint or evaluation call, which distinguishes it from the two siblings. It stops short of five because 'route' and 'admit' are internal jargon that a fresh agent may not map to a concrete outcome.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit when-to-use guidance, no statement of prerequisites (for example that a budget snapshot must be assembled by the caller), and no mention of when not to call it. The only usage-adjacent signal is the failure behavior, which tells the agent what happens on bad input, not when this tool is the right choice.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 3 tool updatesv0.2.0
    • First observeddedup_and_lint_dossier
    • First observedevaluate_agent_response
    • First observedroute_agent_seat

TDQS

A3.5/5.0

Scored across 3 tools

Disambiguation4/5

Each tool targets a distinct phase: seat/budget routing, single-response contract checking, and URL/dossier hygiene. The only wrinkle is dedup_and_lint_dossier bundling two operations with 'and/or' semantics, which could make an agent unsure whether it must supply both inputs.

Naming Consistency5/5

All three follow a clean verb_noun (or verb_noun_noun) snake_case pattern: route_agent_seat, evaluate_agent_response, dedup_and_lint_dossier. Predictable and readable throughout.

Tool Count4/5

Three tools is thin but matches a narrow policy/eval-enforcement role where each tool is a distinct gate. Nothing feels redundant or padded.

Completeness3/5

Covers routing/admission, response validation, and dossier hygiene, but there is no way to inspect or refresh the budget snapshot, enumerate valid roles/contracts, or manage schema definitions—agents hitting an 'unknown role' or 'fails closed' path have no recovery tool.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables agents to route goals across multiple independent APIs, enforce constraints, reject invalid evidence, and return verified outcomes with tamper-evident receipts.
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Enables autonomous AI agents to proactively poll event queues, evaluate document compliance, retrieve grounded knowledge via semantic RAG, enforce human-in-the-loop approval gates, and track multi-model cost budgets.
    10
    MIT
  • A
    license
    C
    quality
    A
    maintenance
    Persistent multi-agent work graph and document-state machine for contracts, specs, slices, evidence, verification, handoffs, and cost-aware AI execution.
    7
    Apache 2.0