agent-eval-policy
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| PORT | No | Port for the HTTP transport. `npm run start:http` serves MCP over HTTP on 127.0.0.1:3333 by default; override with the PORT environment variable. Not needed for the stdio transport (`npm start`), which requires no configuration. | 3333 |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| route_agent_seatB | Route a worker role to a cost tier and admit its planned spend against the budget snapshot. Fails closed on unknown roles, modes, or malformed accounting values. |
| evaluate_agent_responseB | Check one agent response against a contract: anchor line, JSON payload schema, forbidden claims, and verified source-slice citations. |
| dedup_and_lint_dossierA | Canonicalize and deduplicate URLs, and/or lint a markdown dossier against the dossier schema. ok is false on any duplicate, invalid URL, or lint violation. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
| tool-contracts | Each tool description and its JSON input schema, in markdown. |
| eval-fixtures | The fixture file replayed by the native harness and the Promptfoo config. |
| eval-scorecard | Output of the last `npm run eval`. Absent until the harness has run. |
TDQS
Scored across 3 tools
Each tool targets a distinct phase: seat/budget routing, single-response contract checking, and URL/dossier hygiene. The only wrinkle is dedup_and_lint_dossier bundling two operations with 'and/or' semantics, which could make an agent unsure whether it must supply both inputs.
All three follow a clean verb_noun (or verb_noun_noun) snake_case pattern: route_agent_seat, evaluate_agent_response, dedup_and_lint_dossier. Predictable and readable throughout.
Three tools is thin but matches a narrow policy/eval-enforcement role where each tool is a distinct gate. Nothing feels redundant or padded.
Covers routing/admission, response validation, and dossier hygiene, but there is no way to inspect or refresh the budget snapshot, enumerate valid roles/contracts, or manage schema definitions—agents hitting an 'unknown role' or 'fails closed' path have no recovery tool.