Autoeval

Demo

Quickstart, validate the setup, and see passing and failing release gates.

Autoeval is an open-source CLI and MCP server that turns committed evaluation files into versioned, gated checks on AI-agent behavior, scored by the Plumloom backend. It runs from your terminal, your CI, or your coding agent.

Related MCP server: AI Workbench MCP

What you can evaluate

Where it runs

Eval files and the suite manifest are the only inputs you author and commit. Autoeval submits runs to Plumloom, reads back scored results, and the gate turns them into a CI pass/fail.

Quickstart

Requirements: Node.js 22.13+ and a Plumloom CLI key (pl_sk_…). pnpm 11 only if building from source.

1. Connect an AI provider (OpenAI or Together.ai).

2. Create a Plumloom API key.

npm install --global @plumloom/cli
autoeval login
autoeval quickstart

quickstart resolves a workspace and judge model, grades a bundled sample end to end, and prints the result. Autoeval connects to https://api.plumloom.ai automatically. → Quickstart guide

pnpm install --frozen-lockfile && pnpm build

In the examples below, replace autoeval with node packages/cli/dist/cli.js when running from this repository.

File-driven evaluation

An evaluation file uses the configured-run JSON accepted by autoeval eval validate. Artifact and reference-document paths resolve relative to the evaluation file.

{
  "methodology": { "judgeModelId": "<judge-model-id>", "evaluatorInstructions": "..." },
  "configuration": {
    "contextName": "Support conversation",
    "contextType": "conversation",
    "artifactFile": "./data/conversation.json",
    "referenceDocumentFiles": ["./policy/refunds.md"]
  }
}

autoeval eval validate --input support.autoeval.json
autoeval eval create-from --workspace <workspace-id> --input support.autoeval.json --run

Model-ID fields take UUIDs; copy enabled IDs from autoeval models. → Write and run your own evaluation file

Gate a release in CI

A suite manifest names a workspace, lists evaluation files, and declares release thresholds.

workspace: <workspace-id>
gate:
  minOverall: 4.0
  minScenario: 3.5
evals:
  - ./evals/refund-policy.autoeval.json
  - file: ./evals/support-conversation.autoeval.json
    gate:
      metrics:
        factuality: 4.0
        relevance: 3.8

autoeval doctor     --manifest autoeval.suite.yaml   # read-only pre-flight
autoeval suite run  --manifest autoeval.suite.yaml   # run, no release judgment
autoeval suite gate --manifest autoeval.suite.yaml   # apply thresholds

Commit the manifest, every referenced eval file, and any artifacts or reference documents those evals load. Start from examples/suite/autoeval.suite.yaml and its fixtures under examples/smoke/fixtures/, then use autoeval workspace list and autoeval models to replace the synthetic IDs.

Copy .github/workflows/autoeval-suite-gate.yml.example into your repository for a ready-to-use workflow. See Gate a release in CI and Release gating for CI.

For same-repository pull requests, the example workflow also updates one context-aware Autoeval Reliability Report comment per suite and matrix job, showing the gate verdict alongside available Standard, Judgment, and Repeatability evidence. Comment failures never replace the gate verdict or exit code. Fork pull requests are excluded because they do not receive repository secrets.

autoeval workspace list
autoeval models
autoeval results <evaluation-id> <run-id>
autoeval --json results <evaluation-id> <run-id>
autoeval feedback <evaluation-id> <run-id>
autoeval --json feedback <evaluation-id> <run-id>

See the command guide, CLI reference, developer guide, and .env.example. Smoke workflows: examples/smoke/ · sample configs: examples/evals/.

Import OpenAI evaluations

Add your OpenAI key once in the Plumloom UI under AI Providers & Models. The CLI uses only your existing Plumloom CLI key; it never requests an OpenAI key or calls OpenAI directly.

autoeval import status
autoeval import connect
autoeval import review --search rubric
autoeval import run --all-new
autoeval import list
autoeval import insights --workspace <workspace-uuid>
autoeval import show <plumloom-eval-uuid> --show-outputs

Imports create a Plumloom copy and never write back to OpenAI. See the OpenAI import guide for every command, exit code, safety limit, and local/staging verification commands. This import surface is CLI-only.

Detail results use --page and --size (sent as backend page_size). Imported-run histories show the backend's run dates and result totals. Removal reports success only after the backend confirms deleted: true.

Human detail output uses numbered runs and shows unknown for missing model names. Add --json for the full redacted record, including raw_json. Result rows come from the backend's nested results.items page. Omit --run for the newest run, or use the Plumloom run UUID from the run history, not an eval_... source evaluation ID.

Insights use spaced sections and readiness markers. The human flaky-graders panel shows only backend-flagged entries; JSON retains the full insights envelope and grader evaluation IDs.

Interactive import review, import run, import show, and import evals display loading progress on stderr; --json stays machine-readable. Import runs show batch progress for IDs, text files, and all-new selections. Expanded detail output uses terminal-width input/expected/output columns with wrapped content. Ctrl+C cancels review without displaying a late response.

Use it from your coding agent

MCP server — a local stdio server exposing the same action layer as the CLI. Credentials resolve from AUTOEVAL_API_KEY, then the OS credential store; it never prompts. → MCP guide · Codex & Claude Code

Claude Code plugin

claude plugin marketplace add AlchemyInCode/plumloom-autoeval-oss
claude plugin install plumloom-autoeval@plumloom

Gemini CLI extension (skill-only for now: install and trigger supported, end-to-end execution not yet)

gemini extensions install https://github.com/AlchemyInCode/plumloom-autoeval-oss

→ Agent skill guide · Security and permissions

Free Autoeval Pilot

Documentation

Support

• Privacy policy: Plumloom Privacy Policy

• Support: support@plumloom.ai