@cyanheads/evals-mcp-server

Tools

Nine tools for authoring eval records — the draft loop (create, revise, discard, submit), the standalone deterministic checker, and read/list/export:

evals_describe_schema

Return what a record of a given task_type needs before you draft it.

• Static — derived from the record and grader Zod schemas, no disk or runtime state

• Per-type gold shape, appropriate grader kind(s), required/optional fields, and authoring notes

• task_type is one of numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, free_response

evals_create_draft

Create and persist a draft eval record, then reflect it back as a review forcing function.

• Validates against the task_type discriminated union (per-type rules: mcq requires choices; free_response requires an llm_rubric grader)

• Runs a cheap self-consistency check — grader vs gold and each positive must PASS, vs each negative must REJECT

• Returns the normalized record parroted back behind a divider, a per-field review protocol, a ready-to-paste verification subagent prompt, and what's still required before submit

• Optional draft-time verification block and captures (EvalsIDs) when you already hold provenance

• Stays draft — passing self-consistency proves the grader discriminates, not that the gold is right

evals_revise_draft

Surgically patch a draft so each change stays legible.

• Explicit set (dotted-path → value), append (dotted-path → array items), and unset (dotted paths) operations — not full-record rewrites

• Returns the updated record, an itemized list of what changed, and a re-run self-consistency verdict

• Re-validates the full shape and cross-field constraints after the patch

• Draft-only — submitted records are frozen; task_type cannot be patched (start a new draft to change the discriminant)

evals_run_check

Run a grader against one or more candidates without touching a saved record.

• PASS/REJECT per candidate plus the resolved comparison value (e.g. the math.js-evaluated numeric target), so you see why each matched or missed

• candidates accepts strings, numbers, objects, or arrays — whatever the grader kind expects

• Supply gold for gold-relative kinds (exact_match); it is a no-op for target-embedding kinds like numeric and mcq

• llm_rubric cannot run here — it is graded at submit via sampling

evals_submit_draft

Finalize a draft through the committability gate, then freeze it.

• The gate runs the grader against the gold (must PASS), requires ≥1 declared negative case to be REJECTED, and requires a recorded, decorrelated independent verification that agrees with the gold

• Resolves and embeds any captures from EVALS_CAPTURE_DIR, cross-checking the gold against the authoritative captured value

• Rejects duplicates (same content_hash already submitted)

• On pass, flips the record to submitted, stamps submitted_at and a checksum, and freezes it; otherwise refuses with a typed error and the record stays a draft

• free_response llm_rubric is judged via sampling when the client supports it, else admitted on recorded verification alone and flagged server_verified: false

evals_export_records

Compile submitted records to a downstream eval format.

• jsonl (lossless, the lingua franca), csv (a flattened, lossy spreadsheet summary), inspect (UK AISI Inspect AI), lm-eval (EleutherAI lm-evaluation-harness)

• Optional domain / task_type / tag filter

• Only submitted records are exported — drafts are skipped

• Writes the artifact under exports/ and returns the file path, record count, byte size, and a short preview instead of dumping inline

Related MCP server: mcp-eval-harness

Resource

All record data is also reachable through the tool surface — evals_get_record for a single record, evals_list_records to browse. The resource is a convenience mirror for clients that support resources, not the access path.

Features

Built on @cyanheads/mcp-ts-core:

• Declarative tool and resource definitions — single file per primitive, framework handles registration and validation

• Unified error handling — handlers throw, framework catches, classifies, and formats

• Typed error contracts — tools declare their domain failures (reason + recovery), surfaced to the agent

• Pluggable auth: none, jwt, oauth

• Structured logging with optional OpenTelemetry tracing

• STDIO and Streamable HTTP transports

Eval authoring:

• A draft → review → surgical-revise → submit loop, with the server acting as both scribe (normalize, persist, compile) and adversarial checker (run the record's own grader, reject what doesn't hold up)

• Records are a Zod discriminatedUnion keyed on task_type — numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, free_response

• A typed grader DSL serialized with each record — deterministic kinds (numeric via math.js, exact_match, set_match, regex, mcq, json_match) run server-side; llm_rubric routes to sampling

• An enforced committability gate at submit: the gold must pass its own grader, ≥1 negative must be rejected, and a recorded decorrelated verification must agree with the gold

• Optional fleet grounding via the captures EvalsID field — link framework-written tool-call dumps, resolved from EVALS_CAPTURE_DIR and cross-checked against the gold (no server-to-server calls)

• Plain JSON files under EVALS_DATA_DIR — inspectable, diffable, version-controllable records

• Compile to JSONL, CSV, Inspect AI, and lm-evaluation-harness formats

Agent-friendly output:

• The two instructional tools (evals_create_draft, evals_revise_draft) carry the loop's review mechanism in their responses — the parsed record parroted back, a per-field review protocol, and a ready subagent prompt

• Self-consistency verdicts on every draft and revise — per-positive and per-negative results, not just a boolean

• evals_list_records discloses truncation when the limit is hit, so a partial set is never mistaken for the whole corpus

• The submit gate refuses with a typed reason + recovery hint, so a rejected record tells the agent exactly what to fix

Getting started

Add the following to your MCP client configuration file. Set EVALS_DATA_DIR to a writable folder — the server manages drafts/, submitted/, and exports/ under it.

{
  "mcpServers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "bunx",
      "args": ["@cyanheads/evals-mcp-server@latest"],
      "env": {
        "MCP_TRANSPORT_TYPE": "stdio",
        "MCP_LOG_LEVEL": "info",
        "EVALS_DATA_DIR": "/absolute/path/to/evals-data"
      }
    }
  }
}

Or with npx (no Bun required):

{
  "mcpServers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@cyanheads/evals-mcp-server@latest"],
      "env": {
        "MCP_TRANSPORT_TYPE": "stdio",
        "MCP_LOG_LEVEL": "info",
        "EVALS_DATA_DIR": "/absolute/path/to/evals-data"
      }
    }
  }
}

Or with Docker:

{
  "mcpServers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-e", "MCP_TRANSPORT_TYPE=stdio",
        "-e", "EVALS_DATA_DIR=/data",
        "-v", "evals-data:/data",
        "ghcr.io/cyanheads/evals-mcp-server:latest"
      ]
    }
  }
}

For Streamable HTTP, set the transport and start the server:

MCP_TRANSPORT_TYPE=http MCP_HTTP_PORT=3010 EVALS_DATA_DIR=./evals-data bun run start:http
# Server listens at http://localhost:3010/mcp

Prerequisites

• Bun v1.3.2 or higher (or Node.js v24+).

• A writable directory for EVALS_DATA_DIR. No external API key is required.

Installation

1. Clone the repository:

git clone https://github.com/cyanheads/evals-mcp-server.git

2. Navigate into the directory:

cd evals-mcp-server

3. Install dependencies:

bun install

4. Configure environment:

cp .env.example .env
# edit .env and set EVALS_DATA_DIR

Configuration

All server configuration is validated at startup via Zod schemas in src/config/server-config.ts.

See .env.example for the full list of optional overrides.

Running the server

Local development

• Build and run:

  # One-time build
  bun run rebuild
  
  # Run the built server
  bun run start:stdio
  # or
  bun run start:http

• Run checks and tests:

  bun run devcheck   # Lint, format, typecheck, security
  bun run test       # Vitest test suite
  bun run lint:mcp   # Validate MCP definitions against spec

Docker

docker build -t evals-mcp-server .
docker run --rm -e MCP_TRANSPORT_TYPE=stdio -e EVALS_DATA_DIR=/data -v evals-data:/data evals-mcp-server

The Dockerfile defaults to HTTP transport, stateless session mode, and logs to /var/log/evals-mcp-server. OpenTelemetry peer dependencies are installed by default — build with --build-arg OTEL_ENABLED=false to omit them.

Project structure

Development guide

See CLAUDE.md/AGENTS.md for development guidelines and architectural rules. The short version:

• Handlers throw, framework catches — no try/catch in tool logic

• Use ctx.log for request-scoped logging; records persist to disk via the record-store service, not ctx.state

• Register new tools and resources in the createApp() arrays in src/index.ts

• The server is the source of truth — validate inputs, run the grader as a hard gate, and never admit a record on assertion alone

Contributing

Issues and pull requests are welcome. Run checks and tests before submitting:

bun run devcheck
bun run test

License

Apache-2.0 — see LICENSE for details.