Plumloom Autoeval

The release-confidence layer for AI-agent changes.

Autoeval is an open-source CLI and MCP server that turns committed evaluation files into versioned, gated checks on AI-agent behavior, scored by the Plumloom backend. It supports scenario, conversation, and agent-trace evaluation contexts from development through CI.

LLM outputs and judge scores can vary between runs. For scenario evaluations, Autoeval supports multiple trials and reports reliability evidence such as confidence intervals and consistency statistics. This helps developers judge whether a release decision is supported by evidence beyond a single score.

Architecture

Eval files and the suite manifest are the only inputs you author and commit; Autoeval submits runs to Plumloom, reads back scored results, and the gate turns them into a CI pass/fail.

Related MCP server: AI Workbench MCP

Requirements

• Node.js 22.13 or newer

• A Plumloom CLI key beginning with pl_sk_

pnpm 11 is needed only to build from source.

Install

npm install --global @plumloom/cli

To build from source, run pnpm install --frozen-lockfile && pnpm build. In the examples below, replace autoeval with node packages/cli/dist/cli.js when running from this repository.

Configure Plumloom

1. Connect an AI provider so Plumloom can run evaluations. OpenAI and Together.ai are supported.

2. Create a Plumloom API key for authentication from your terminal, CI, or development tools.

Autoeval connects to https://api.plumloom.ai automatically. No API URL configuration is required.

Quickstart

autoeval login
autoeval quickstart

quickstart resolves a workspace and judge model, grades a bundled sample end to end, and prints the result. See the quickstart guide.

File-driven evaluation

An evaluation file uses the configured-run JSON accepted by autoeval eval validate. Artifact and reference-document paths resolve relative to the evaluation file.

{
  "methodology": { "judgeModelId": "<judge-model-id>", "evaluatorInstructions": "..." },
  "configuration": {
    "contextName": "Support conversation",
    "contextType": "conversation",
    "artifactFile": "./data/conversation.json",
    "referenceDocumentFiles": ["./policy/refunds.md"]
  }
}

autoeval eval validate --input support.autoeval.json
autoeval eval create-from --workspace <workspace-id> --input support.autoeval.json --run

Model-ID fields take UUIDs; copy enabled IDs from autoeval models. See Write and run your own evaluation file for the full walkthrough.

Add a CI release gate

A suite manifest names a workspace, lists evaluation files, and declares release thresholds. Paths resolve relative to the manifest.

workspace: <workspace-id>
gate:
  minOverall: 4.0
  minScenario: 3.5
evals:
  - ./evals/refund-policy.autoeval.json
  - file: ./evals/support-conversation.autoeval.json
    gate:
      metrics:
        factuality: 4.0
        relevance: 3.8

Commit the manifest, every referenced eval file, and any artifacts or reference documents those evals load. Start from examples/suite/autoeval.suite.yaml and its fixtures under examples/smoke/fixtures/, then use autoeval workspace list and autoeval models to replace the synthetic workspace and model IDs in your own copies.

autoeval doctor --manifest autoeval.suite.yaml
autoeval suite run --manifest autoeval.suite.yaml
autoeval suite gate --manifest autoeval.suite.yaml

doctor checks credentials, files, workspace access, and model IDs without creating resources. suite run executes the listed evaluations without a release judgment. suite gate applies the manifest thresholds: PASS=0, FAIL/INCONCLUSIVE=1, and ERROR=5.

Copy .github/workflows/autoeval-suite-gate.yml.example into your repository for a ready-to-use adopter workflow. See Gate a release in CI and Release gating for CI for setup and policy details.

Common commands include:

autoeval workspace list
autoeval models
autoeval results <evaluation-id> <run-id>
autoeval --json results <evaluation-id> <run-id>

The command guide and CLI reference cover the full command surface. For environment and credential behavior, see the developer guide and .env.example. Runnable smoke workflows live under examples/smoke/, additional sanitized configurations live under examples/evals/, and reliability semantics are documented in the documentation home.

MCP server

Autoeval includes a local stdio MCP server that exposes the same action layer as the CLI. It resolves credentials from AUTOEVAL_API_KEY and then the OS credential store, and it never prompts. See the MCP server guide and Autoeval with Codex and Claude Code.

Agent skill

Install the package from the Claude Code marketplace:

claude plugin marketplace add AlchemyInCode/plumloom-autoeval-oss
claude plugin install plumloom-autoeval@plumloom

Install the Gemini CLI extension:

gemini extensions install https://github.com/AlchemyInCode/plumloom-autoeval-oss

Gemini CLI is skill-only for now: install and trigger are supported, but end-to-end Autoeval execution is not. See the agent skill guide and agent skill security and permissions.

• Privacy policy: Plumloom Privacy Policy

• Support: support@plumloom.ai

Documentation

• Documentation home

• Quickstart

• Command guide

• Deterministic CLI developer guide

• Deterministic command reference

• MCP server guide

• Codex and Claude Code integration

• Agent skill for coding agents

• Agent skill security and permissions

• DeepSeek Harness integration

• Release gating for CI

• Troubleshooting

• Security policy

• Contributing