evalmine

Leaderboards do not tell you whether a model change improves the work you actually do. evalmine turns controlled evidence from your own tasks into a decision you can defend. It accepts direct model suites, isolated agent episodes, or completed artifacts from an application-owned harness; verifies their provenance; builds blind review; and calibrates an LLM judge against human labels. If the judge has not earned trust, evalmine refuses to present its result as headline-ready.

The project is local-first, source-installed, and pre-release. The CLI and MCP server call the same guarded library operations.

The recording uses the deterministic fake adapter: no provider call, key, or spend.

What it does

A controlled workflow runner is also available for reproducible fixture, fan-out, and artifact jobs. Outputs stay local; experiment, import, and workflow evidence is hash-verifiable, and review or decision reports are self-contained HTML.

Direct adapters are included for Anthropic, OpenAI, Google, and OpenRouter. A deterministic fake adapter exercises the complete suite and reporting path without network access.

Related MCP server: Coval MCP Server

Quick start

git clone https://github.com/hishamalward/evalmine.git
cd evalmine
python3 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev,mcp]"

evalmine validate examples/everyday-eight.yaml
evalmine run examples/everyday-eight.yaml \
  --models anthropic/claude-haiku-4-5,google/gemini-2.5-flash \
  --fake

Python 3.10 or newer is required. The fake run is deterministic and spends nothing. Its report demonstrates the central rule: a win rate remains visibly uncalibrated when human ↔ judge agreement is below the suite's floor.

For a real run, export only the provider credentials you need and set an explicit cap:

export ANTHROPIC_API_KEY=...
export GOOGLE_API_KEY=...

evalmine run examples/everyday-eight.yaml \
  --models anthropic/claude-haiku-4-5,google/gemini-2.5-flash \
  --max-cost 0.50

Use OPENAI_API_KEY for openai/... models and OPENROUTER_API_KEY for openrouter/... models. OpenRouter is a dedicated adapter: nested catalog slugs, provider pins, response-reported charges, and routing options remain part of the evidence and cache identity.

Agent experiments

A version-2 manifest pins a seed repository and declares the arms, episodes, repeats, isolation policy, configuration treatments, validators, and evaluation method.

evalmine experiment validate examples/agent-model-comparison.yaml
evalmine experiment plan examples/agent-model-comparison.yaml
evalmine experiment prepare examples/agent-model-comparison.yaml \
  --out /tmp/evalmine-runs

evalmine experiment preflight <prepared-dir>
evalmine experiment execute <prepared-dir> --allow-provider-calls
evalmine experiment check <prepared-dir>
evalmine experiment report <prepared-dir>

Preparation and preflight launch no agents. Execution uses the authentication already owned by Claude Code, Codex CLI, or Gemini CLI. Each arm receives an isolated workspace and fresh session; subsequent turns resume only within that run. Judging and decisions are separate, explicit stages—see the experiment lifecycle.

External artifacts and the harness kit

Use the external lane when generation or application data must remain outside EvalMine. The producer supplies a sanitized evalmine-import.yaml plus hash-pinned JSONL:

evalmine experiment import examples/external-artifacts \
  --out /tmp/evalmine-external
evalmine experiment verify /tmp/evalmine-external
evalmine experiment report /tmp/evalmine-external

Import validates the schema and comparison grid, preserves source-file and line provenance, and makes zero model calls. EvalMine does not query the producer's database and does not pretend to sanitize arbitrary application data: sanitization is the producer's boundary.

TypeScript producers can use @evalmine/harness-kit to build and preflight a bundle. The kit accepts completed records only. It generates nothing, judges nothing, has no ledger or database access, and has no runtime dependencies. Until its first registry release, consume an exact-commit source snapshot or an npm pack tarball.

Controlled workflows

Workflow manifests coordinate direct-argv jobs, frozen fixtures, dependencies, and captured artifacts without weakening the direct-API cost boundary.

evalmine workflow plan examples/music-backoff-workflow.yaml
evalmine workflow run examples/music-backoff-workflow.yaml \
  --out /tmp/evalmine-workflows --allow-commands

Provider-marked nodes need a second provider-call gate. Arbitrary direct-API shell nodes are refused; use a cost-capped suite for those calls.

MCP for Claude Code and other agents

evalmine-mcp exposes suite, experiment, external-import, and workflow operations over stdio. To register it in a Claude Code project:

pip install -e ".[mcp]"
cp .mcp.json.example .mcp.json

Restart the MCP client after adding the file. The example keeps access inside ${CLAUDE_PROJECT_DIR:-.} and starts with provider calls, validator commands, external writes, and workflow commands disabled. A mutating or spending operation requires both the tool-call argument and its matching EVALMINE_MCP_ALLOW_* server gate. Provider keys are inherited from the launch environment and are not written into .mcp.json.

See the MCP contract for the tool list, path containment, cost ceilings, and structured refusal behavior.

Safety and evidence rules

• Live direct calls are planned and cost-estimated before the first request. Over-cap runs are refused; missing cost is never silently reported as $0.

• Suite placeholders, unknown fields, model prices, and key-shaped literals are validated before execution.

• Provider calls, validator commands, workflow commands, and external writes have separate authorization gates.

• Experiment, import, and workflow evidence is create-once, hash-pinned, bounded, and independently verifiable. Credentials are removed or redacted from captured runner and workflow output where supported.

• External bundles must already be sanitized. They intentionally contain the prompts and outputs needed for review, so producers must not include private source data unnecessarily.

• Reports keep condition identity hidden during review and reveal it only in decision evidence.

• Judge agreement, position sensitivity, schema pass rate, objective checks, latency, and cost remain separate signals. A failed calibration gate cannot produce a clean headline.

Status

Version 0.1.0 is alpha software. The suite engine, four live adapters, isolated agent experiments, external import, workflow runner, TypeScript harness kit, blind reports, calibrated decisions, and guarded MCP surface are implemented and covered by network-free tests across Linux, macOS, and Windows.

The remaining release proof is operational: a representative real evaluation must be human-labeled, pass its calibration gate, be inspected, and produce the first human-owned DECISIONS.md entry. EvalMine is not a hosted UI, RAG evaluator, fine-tuning system, or product database connector.

Documentation

• Specification — normative product and safety contract

• How it works — implementation tour

• Harness kit — producer-side TypeScript API

• Examples — synthetic suites, experiments, imports, and workflows

• Contributing — development workflow

This README describes the present product surface. Files under docs/plans/ record design and implementation history; they are context, not the current user manual. Where wording differs, the specification wins.

Development

pip install -e ".[dev,mcp]"
python -m pytest -q
python -m ruff check src tests
npm test --prefix packages/harness-kit

CI runs the complete Python suite on Linux, macOS, and Windows with Python 3.10 and 3.13, plus the TypeScript round trip and a repository-history secret scan.

License

MIT. See LICENSE.