verify-mcp

An MCP server for verify, the VeriHarness verification harness. It lets an agent in Claude Code, Codex or Cursor verify a task, run a benchmark, and read the results through ten tools.

verify-mcp is not published to npm or to a public plugin marketplace. Install it from this repository.

Contents

• What verify does

• Requirements

• Quick start in Claude Code

• Task workspace

• Workflows

• Tools

• Configuration

• Other hosts

• Windows

• Security

• Protocol

• Development

• License

Related MCP server: mcp_server_for_claudes_toolbox

What verify does

verify takes one task and the results of N independent attempts at it (the rollouts). Each attempt comes from a Worker. A verifier model then checks the rollouts against the task's own files in four phases. Each phase has a role:

1. Checker: resolve disagreements. One session finds the claims on which the rollouts differ, runs the check that separates them, and records who was right (ledger_elim.json).

2. Challenger: challenge the consensus. A second, isolated session finds the claims that all rollouts share and tries to break each one against the inputs (ledger_fals.json).

3. Reviewer: adjudicate. A fresh session reads only the two records. It selects the best rollout as the base, lists the changes the evidence supports, and lists the questions it cannot settle (finish.json).

4. Fixer: repair. The Reviewer session applies those changes and delivers the result to out/deliverables/, with a record of what it changed (repair.json).

The verifier is not a stronger judge. Its advantage comes from the structure of the rollout pool and from evidence that it collects from the task's files. The method, the benchmarks and the results are described in the verify README and in the paper.

verify-mcp does not change the method. It starts the verify command-line tool (veriharness), reports progress while it runs, and returns its output.

Requirements

• Bun 1.1 or later on PATH. CI uses Bun 1.4.2.

• A checkout of danielsimonjr/verify at version 0.12.15 (@danielsimonjr/verify, the pinned version) or later, with its dependencies installed (bun install). The Claude Code plugin expects it at ~/Github/verify. The Claude Code provider needs 102894a or later; a checkout from 756bc2b up to that commit runs local models only.

• verify's agent runtime, pi, installed in that checkout: run harness/scripts/setup_pi.sh once. It installs the pinned pi (@danielsimonjr/pi) into harness/vendor/. verify_driver and verify_runner need it for every provider except claude-code.

• A model for the verifier:

  • Claude Code: the claude program, version 2.1 or later, signed in on the host. verify uses the login that Claude Code holds (a subscription or ANTHROPIC_API_KEY). See Claude Code as the verifier; or

  • a local server: Ollama on http://127.0.0.1:11434, or llama.cpp's llama-server on http://127.0.0.1:8080; or

  • a hosted provider that pi supports, with its credentials set in the environment. The verify README lists them.

• For benchmark runs only: Docker for the graders, and the benchmark archive (VERIHARNESS_BENCH_ROOT). See the verify README.

Quick start in Claude Code

1. Clone verify and install its dependencies and pi runtime:

   git clone https://github.com/danielsimonjr/verify.git ~/Github/verify
   cd ~/Github/verify
   bun install
   harness/scripts/setup_pi.sh

2. Install the plugin:

   claude plugin marketplace add danielsimonjr/verify-mcp
   claude plugin install verify-mcp@verify-mcp

   In a session, the same steps are /plugin marketplace add danielsimonjr/verify-mcp and /plugin install verify-mcp@verify-mcp. To load a local checkout instead, start Claude Code with claude --plugin-dir /path/to/verify-mcp/plugin.

3. In a session, ask Claude to call verify_status. The result shows the verify command that the server resolved, the data and runs directories, and whether this verify build has model-check.

4. Check the model before a long run. For a local Ollama model, ask Claude to call verify_model_check with these arguments:

   { "provider": "ollama", "model": "qwen2.5-coder:7b" }

   For Claude Haiku through Claude Code, use { "provider": "claude-code", "model": "claude-haiku-4-5-20251001" }.

5. Verify one task workspace (see Task workspace) with verify_driver:

   {
     "task_dir": "/path/to/task",
     "provider": "ollama",
     "model": "qwen2.5-coder:7b",
     "env": "none"
   }

   env: "none" runs without verify's jail. Use it on Windows and macOS, where the jail is not available, and always with claude-code. See Windows and Security.

6. Read the decision with verify_read_result:

   { "task_dir": "/path/to/task", "artifact": "finish" }

   Then read the list of delivered files with "artifact": "deliverables".

The plugin also adds the verify skill and the /verify command, which tell the agent the order in which to call the tools.

Task workspace

The verifier sees one directory. You make it yourself, or verify_materialize builds it from a benchmark archive:

<task>/
  spec/task.md                  the task description
  workspace/                    the task's input files
  rollouts/<name>/deliverables/ what that rollout delivered
  rollouts/<name>/trajectory/   its execution record (optional)

verify_driver refuses a directory that has no rollouts/. Rollout names are directory names, for example r01 and r02.

The driver writes its output into the same directory:

The driver also writes the model transcripts under session/. With contract: "pick-only", the driver stops after the Reviewer and writes no repair.json or out/deliverables/.

A batch run has the same files for each task, under VERIHARNESS_RUNS/<run>/<bench>_<pool>/. The cell directory also holds run.json, and verify_score writes scores.json and scores.partial.jsonl there.

Workflows

Verify one task

1. verify_status: confirm the verify command and the directories.

2. verify_model_check: confirm that the model server is up, the model is present, and the model can call tools. verify refuses a model that cannot call tools, because a verifier that cannot call tools writes no ledger.

3. verify_driver with task_dir, provider, model and, off Linux or with claude-code, env: "none".

4. verify_read_result with task_dir and finish, then repair and deliverables.

Run a benchmark

The benchmarks are apex, wsb, wb, sb2 and jb. A cell is one bench:pool pair, for example wb:flash. A pool named after a lane runs on that lane: fable, opus, haiku or sonnet. The archived flash pools (Gemini 3.5 Flash rollouts) run on the fable lane. For another pool name, set lane.

1. verify_materialize with bench: build the task workspaces from the benchmark archive.

2. verify_runner with cells and run_name: verify every task in each cell. The runner can resume. It skips finished tasks and restarts unfinished ones.

3. verify_score with cell_dir: score the cell against the archived scores and re-grade the delivered bundles.

4. verify_list_runs and verify_read_result with run, cell and an artifact.

verify_grade grades one deliverables directory. verify_env_derive builds the WorkBuddy images that the native environments use.

Tools

Defaults

The server passes an optional flag only when you set it, so verify keeps its own defaults:

When you set no provider and model, verify passes none to pi, and pi uses its own default.

Local providers are ollama (default http://127.0.0.1:11434) and llamacpp, with the aliases llama.cpp and llama-cpp (default http://127.0.0.1:8080). Both need model. base_url overrides the address. The provider field also accepts the hosted providers that verify accepts.

Claude Code as the verifier

Provider claude-code runs each verifier turn as claude -p with Claude Haiku or Claude Sonnet. It uses the login that Claude Code holds and does not read or print a credential.

• model is a full model id: claude-haiku-4-5-20251001 or claude-sonnet-5-5.

• env must be none. The jail replaces $HOME, so Claude Code finds no login in it.

• Do not set base_url, context_size, temperature, max_tokens, top_p or request_timeout for verify_driver or verify_runner, or thinking for verify_driver. verify refuses them with this provider. verify_model_check accepts request_timeout.

• All four runner lanes use this provider: fable is claude-fable-5-1, opus is claude-opus-5-5, haiku is claude-haiku-5-5 and sonnet is claude-sonnet-5-5. A cell such as sb2:haiku runs on the haiku lane. Set env: "none" for the run.

• The fable and opus lanes start two tasks at a time and haiku and sonnet four, because the account's usage limit is shared with every other Claude Code session. lane_max raises a lane, for example { "haiku": 4 }. When the account reaches its usage limit, the runner starts no more tasks on that lane.

A runner call for one cell on Haiku:

{
  "cells": ["sb2:haiku"],
  "run_name": "haiku-trial",
  "env": "none",
  "lane_max": { "haiku": 4 }
}

verify's Claude Code design document gives the flags that each turn uses and what they do not isolate.

Context size, batches and workers

context_size on the main model takes a number or "auto". On a role, on verify_batch and on verify_workers, it takes "auto" or a whole number above 4096. "auto" uses the window that the server reports: a loaded Ollama model, then num_ctx, then llama.cpp's n_ctx. It also overrides VERIHARNESS_CONTEXT_SIZE. verify_model_check returns the window as window and its source as windowSource.

verify_batch splits an items file into task folders. Each folder fits a token budget: batch_tokens, or half the window of the worker model (provider and model). Give one of the two, not both. split is jsonl, blank-line or heading:REGEX. The result holds manifest.json: the budget, the window and its source, and the estimate of each batch. The default model profile applies when the call sets no batch_tokens. shared files are read whole, so the estimate counts them. reference files (verify 0.7.0 or later) are only searched, such as a large CHANGELOG, so the estimate does not count them.

verify_workers runs count rollouts (default 3) on each batch of that folder. Each rollout runs in a temp copy, and its record goes to rollouts/<rollout>/trajectory/worker.json. A local model runs one batch at a time; Claude Code runs up to the lane cap and needs env: "none". Each finished rollout is one progress notice. The result holds rollouts and summary. A rollout with an error makes the call an error, and the records are still returned. A Claude Code usage limit stops the run (exit 75); call again after the limit resets, and the complete rollouts are skipped. max_turns (verify 0.6.0 or later) stops a worker that passes that number of assistant turns, with the error max-turns; use it for a small local model that can loop. schema (a JSON Schema file), retries, nudge_timeout and allow_compaction (verify 0.9.0 or later) check the deliverable shape, repeat a rollout whose output was empty or wrong, ask a stopped pi worker for its answer (once for each kind of failure), and accept a compacted rollout. A pi worker also loads a session guard (verify 0.12.11 or later): it blocks a repeated read, grep, find or ls call and a call with an argument that the tool does not list (verify 0.12.13 or later), stops a session after three turns in which every tool call failed (error stuck, verify 0.12.12 or later), cuts each tool result to the room that is left before compaction, and cuts a final message that has more records than the batch has items; frequency_penalty is not set by default. verify's batching guide has a worked example.

{ "items": "TODO-closed.md", "split": "heading:^### TODO line (\\d+)$", "spec": "task.md",
  "reference": ["CHANGELOG.md"], "prompt": "worker_prompt.md", "out": "work",
  "provider": "ollama", "model": "qwen3.5:9b-64k", "context_size": "auto" }

A model for each role

roles on verify_driver and verify_runner gives a verifier role its own provider and model. The roles are checker, challenger, reviewer and fixer. Each entry needs provider and model. A local (ollama or llamacpp) entry can also set base_url and context_size. The field needs verify 0.4.0 or later. context_size: "auto" needs verify 0.5.0.

• A role left out uses provider and model, or the runner lane.

• A fixer left out uses the reviewer's model and continues its session. A fixer on another model starts a new session.

• Any claude-code role needs env: "none".

• verify refuses base_url or context_size on a role that is not local.

A local Checker and Challenger with Claude Opus 5.5 as the Reviewer and the Fixer:

{
  "task_dir": "C:/runs/demo/sb2_haiku/t1",
  "provider": "claude-code",
  "model": "claude-opus-5-5",
  "env": "none",
  "roles": {
    "checker": { "provider": "ollama", "model": "qwen3.5:9b" },
    "challenger": { "provider": "ollama", "model": "qwen3.5:9b" }
  }
}

verify's roles document gives the session rules and the local-server warning.

Input checks

verify_runner refuses input that verify would act on unsafely:

• run_name and each pool in cells must be one path segment: letters, digits, ., _ and -, not starting with a dot. verify joins both under VERIHARNESS_RUNS and deletes an existing task workspace there.

• lane and each lane_max key must be fable, opus, haiku or sonnet. Each lane_max value must be 1 or more.

• The fable and opus caps are set once: with max_fable or max_opus, or in lane_max. verify applies lane_max over the other two, so a second value would be dropped.

• roles takes only the keys checker, challenger, reviewer and fixer, and each entry needs provider and model. A provider must not contain :, because verify splits PROVIDER:MODEL on the first colon.

• Each cell_cap entry must be key=N, where key is a bench or default and N is 1 or more. verify accepts a cap of 0, and its scheduler then never starts the cell's tasks.

verify_driver, verify_runner, verify_score, verify_materialize and verify_workers carry destructiveHint: true, because they overwrite or delete files under the task, batch or runs directory.

verify_read_result reads at most max_bytes of a text artifact (default 262144, maximum 2000000). It refuses a path with .., and a symbolic link that leaves the runs directory or the task directory.

Progress and timeouts

Long tools send notifications/progress, at least every 15 seconds. At the timeout, or when the client cancels, the server stops the verify command and every process under it, including the detached processes that the driver starts for each agent turn. The result keeps the last 65536 characters of stdout and of stderr.

timeout_seconds overrides the default for one call.

Errors

Configuration

Environment variables

The plugin ships no default model. Its .mcp.json passes each of these variables through, empty when unset, so a machine sets its own default in the env block of its Claude Code settings.json. Verify refuses a model that does not report a num_ctx of at least the context size, so a default Ollama model needs a num_ctx set, in a variant made with ollama create or by loading the model at that size. A call that names a provider or a model ignores the whole default.

example.mcp.json shows a server entry with every variable set and placeholder values. Copy it to .mcp.json in a project, or into your user .mcp.json, set the paths and the model, and use it when you run verify-mcp without the plugin. Git ignores a .mcp.json at the repository root, so a local copy stays local. With the plugin installed, set the VERIFY_MCP_* variables in the env block of your Claude Code settings.json instead of adding a second server.

The server expands ~ and passes a value that you set to verify as an absolute path.

The server gives verify its own environment. An MCP host can start a stdio server with a reduced environment, so a variable that you export in your shell can be absent in the server. If a hosted provider's key or one of the variables above does not reach verify, set it in the env block of the server's MCP configuration. For the Claude Code plugin, that is plugin/.mcp.json.

How the server finds verify

The server starts the verify command as a child process. It does not import verify as a library: verify's command modules read the environment when they load, write to stdout and stderr, and exit the process, and a driver run starts process trees that must be stopped as a group.

The server looks for the command in this order:

1. VERIHARNESS_BIN. A .ts, .js or .mjs file starts with bun. Any other file runs as it is.

2. node_modules/@danielsimonjr/verify/harness/cli.ts, which bun install in this repository puts there. It is the npm package @danielsimonjr/verify, pinned to the exact version named in src/pin.ts.

With the copy in node_modules, and no VERIHARNESS_DATA or VERIHARNESS_RUNS set, the data and run directories are ./data and ./runs in the server's working directory. That keeps run output out of node_modules. A verify checkout keeps verify's own defaults, which are data/ and runs/ in the checkout.

The Claude Code plugin contains the server but not verify, because verify reads its prompts, skills and scripts from the folders next to its command. To use a checkout at another path, change VERIHARNESS_BIN in plugin/.mcp.json.

Other hosts

Codex

codex plugin marketplace add danielsimonjr/verify-mcp
codex plugin add verify-mcp@verify-mcp

You can also install it from /plugins after you add the marketplace. Codex starts bun ./src/index.ts in the plugin root (mcp.json).

Cursor

Copy the checkout into the local plugin directory. Cursor does not load a symbolic link to a path outside that directory.

mkdir -p ~/.cursor/plugins/local
cp -R /path/to/verify-mcp ~/.cursor/plugins/local/verify-mcp
cd ~/.cursor/plugins/local/verify-mcp && bun install

Restart Cursor, or run Developer: Reload Window. Then open Customize and install the plugin. Cursor can load both the Cursor manifest and the root Agent Plugin manifest, and then shows two verify servers. Keep the one from .cursor-plugin/plugin.json.

Any MCP client

git clone https://github.com/danielsimonjr/verify-mcp.git
cd verify-mcp
bun install
bun src/index.ts              # stdio
bun src/index.ts --http 8787  # Streamable HTTP on http://127.0.0.1:8787/mcp

The HTTP server listens on the loopback address only.

Windows

• verify's default environment, jail, uses Linux mount namespaces. On Windows, pass env: "none" to verify_driver and verify_runner. Without it, the driver refuses to start and says why.

• For claude-code, Claude Code needs Git for Windows. verify starts claude from PATH. If claude is a .cmd shim, set VERIHARNESS_CLAUDE_BIN to the path of claude.exe in the server's environment. Node cannot start a .cmd file without a shell.

• Run harness/scripts/setup_pi.sh from Git Bash. It installs pi with npm, which makes the pi, pi.cmd and pi.ps1 launchers. verify-mcp starts verify with Bun, and Bun starts pi through them.

• The server stops a timed-out verify command with taskkill /T /F, so the processes under it stop too.

• CI runs the test suite on windows-latest and on ubuntu-latest.

Security

• env: "none" is not a sandbox. The verifier runs shell commands as your user, on your machine. Verify only rollouts whose files you are willing to have read and run there. On Linux, the default jail hides the rest of $HOME, the archived scores and the benchmark answer keys from the verifier.

• The Claude Code lanes always run without the jail. A verifier on haiku or sonnet can read the results of other tasks and the benchmark answer keys. Do not use those scores where answer-key secrecy matters. With env: "none", the runner also runs the pi cells of the same run without the jail.

• verify_list_runs and verify_read_result read only inside the runs directory or the task directory that you name. They accept fixed artifact names, not paths.

• verify_runner refuses run and pool names that would reach outside the runs directory.

• The server returns no environment values in tool results. It passes its own environment to verify, so the verifier model's credentials reach verify only through that environment.

Protocol

The server negotiates MCP protocol revision 2026-07-28. It also accepts a client that starts with a 2025 initialize, so hosts that do not yet ask for 2026-07-28 can connect. A test pins a client to 2026-07-28 and checks that the server answers with that revision.

• The server declares tools only. It does not declare logging, roots or sampling, which the 2026-07-28 revision deprecates or replaces.

• Transports: stdio (serveStdio) and Streamable HTTP (createMcpHandler), from @modelcontextprotocol/server 2.3.0. Progress notifications switch an HTTP response to SSE. The older HTTP+SSE transport is not implemented.

• The v1 SDK, @modelcontextprotocol/sdk, does not support 2026-07-28, so the server does not use it.

Development

bun install
bun run typecheck
bun test
bun run bundle   # after a change to src/, commands/ or skills/

bun run bundle rebuilds plugin/. Commit the rebuilt plugin/ with the change: tests/bundle.test.ts fails when plugin/ does not match a fresh build.

CI (.github/workflows/ci.yml) runs bun install --frozen-lockfile, bun run typecheck and bun test on ubuntu-latest and windows-latest.

Source

Plugin files

tests/manifests.test.ts checks each manifest against the host's published schema (copies are in tests/fixtures/) and checks that every manifest carries the package.json version.

License

Apache-2.0, the same license as verify.