Autoeval

Demo

Quickstart, validate the setup, and see passing and failing release gates.

Autoeval is an open-source CLI and MCP server that turns committed evaluation files into versioned, gated checks on AI-agent behavior, scored by the Plumloom backend. It runs from your terminal, your CI, or your coding agent.

Related MCP server: AI Workbench MCP

What you can evaluate

Where it runs

Eval files and the suite manifest are the only inputs you author and commit. Autoeval submits runs to Plumloom, reads back scored results, and the gate turns them into a CI pass/fail.

Quickstart

Requirements: Node.js 22.13+ and a Plumloom CLI key (pl_sk_…). pnpm 11 only if building from source.

1. Connect an AI provider (OpenAI or Together.ai).

2. Create a Plumloom API key.

npm install --global @plumloom/cli
autoeval login
autoeval quickstart

quickstart resolves a workspace and judge model, grades a bundled sample end to end, and prints the result. Autoeval connects to https://api.plumloom.ai automatically. → Quickstart guide

pnpm install --frozen-lockfile && pnpm build

In the examples below, replace autoeval with node packages/cli/dist/cli.js when running from this repository.

File-driven evaluation

An evaluation file uses the configured-run JSON accepted by autoeval eval validate. Artifact and reference-document paths resolve relative to the evaluation file.

{
  "methodology": { "judgeModelId": "<judge-model-id>", "evaluatorInstructions": "..." },
  "configuration": {
    "contextName": "Support conversation",
    "contextType": "conversation",
    "artifactFile": "./data/conversation.json",
    "referenceDocumentFiles": ["./policy/refunds.md"]
  }
}

autoeval eval validate --input support.autoeval.json
autoeval eval create-from --workspace <workspace-id> --input support.autoeval.json --run

Model-ID fields take UUIDs; copy enabled IDs from autoeval models. → Write and run your own evaluation file

Gate a release in CI

A suite manifest names a workspace, lists evaluation files, and declares release thresholds.

workspace: <workspace-id>
gate:
  minOverall: 4.0
  minScenario: 3.5
evals:
  - ./evals/refund-policy.autoeval.json
  - file: ./evals/support-conversation.autoeval.json
    gate:
      metrics:
        factuality: 4.0
        relevance: 3.8

autoeval doctor     --manifest autoeval.suite.yaml   # read-only pre-flight
autoeval suite run  --manifest autoeval.suite.yaml   # run, no release judgment
autoeval suite gate --manifest autoeval.suite.yaml   # apply thresholds

Commit the manifest, every referenced eval file, and any artifacts or reference documents those evals load. Start from examples/suite/autoeval.suite.yaml and its fixtures under examples/smoke/fixtures/, then use autoeval workspace list and autoeval models to replace the synthetic IDs.

Copy .github/workflows/autoeval-suite-gate.yml.example into your repository for a ready-to-use workflow. See Gate a release in CI and Release gating for CI.

For same-repository pull requests, the example workflow also updates one context-aware Autoeval Reliability Report comment per suite and matrix job, showing the gate verdict alongside available Standard, Judgment, and Repeatability evidence. Comment failures never replace the gate verdict or exit code. Fork pull requests are excluded because they do not receive repository secrets.

autoeval workspace list
autoeval models
autoeval results <evaluation-id> <run-id>
autoeval --json results <evaluation-id> <run-id>

See the command guide, CLI reference, developer guide, and .env.example. Smoke workflows: examples/smoke/ · sample configs: examples/evals/.

Use it from your coding agent

MCP server — a local stdio server exposing the same action layer as the CLI. Credentials resolve from AUTOEVAL_API_KEY, then the OS credential store; it never prompts. → MCP guide · Codex & Claude Code

Claude Code plugin

claude plugin marketplace add AlchemyInCode/plumloom-autoeval-oss
claude plugin install plumloom-autoeval@plumloom

Gemini CLI extension (skill-only for now: install and trigger supported, end-to-end execution not yet)

gemini extensions install https://github.com/AlchemyInCode/plumloom-autoeval-oss

→ Agent skill guide · Security and permissions

Free Autoeval Pilot

Documentation

Support

• Privacy policy: Plumloom Privacy Policy

• Support: support@plumloom.ai