Skip to main content
Glama

AI Workbench

License

AI Workbench supervises AI coding agents, captures evidence, validates work, applies acceptance policy, and produces auditable PR-ready reports.

The PyPI package remains ai-workbench-mcp for this public alpha because the ai-workbench package name is already occupied. The product and CLI are AI Workbench:

pip install ai-workbench-mcp
ai-workbench --help

Current source metadata targets unpublished ai-workbench-mcp==0.8.0a0. This public alpha consolidates local supervision, evidence capture, validation, acceptance policy, and PR reporting into one product surface.

Public Alpha Warning

The supervisor is the preferred automated evidence path, but daemon, Codex hook, and OpenCode adapter coverage are alpha mechanisms. AI Workbench checks evidence quality and acceptance readiness; it does not prove the work is absolutely correct. High-risk work still requires human review.

Related MCP server: Spec Kit Acceptance Gate MCP

Architecture

  • AI Workbench supervisor captures local evidence.

  • AI Workbench validation writes validation_report.json.

  • AI Workbench quality gate writes revision_decision.json.

  • AI Workbench PR/report surfaces render accept, needs_review, or block.

Agent output is a proposal. Workbench accepts evidence.

MCP is the connection protocol. AI Workbench MCP is the tool server. Acceptance is decided by the selected validation profile and quality gate. The agent performs. Workbench accepts. MCP connects them.

Quick Start

Register a project once and start the local supervisor:

pip install ai-workbench-mcp
ai-workbench supervisor setup --project-dir . --task-type code_change
ai-workbench supervisor start

Run Codex, OpenCode, Goose, or another supported local workflow in the project. Then inspect the latest report:

ai-workbench supervisor status
ai-workbench reports show latest --project-dir .

Render PR-ready artifacts from a finalized run:

ai-workbench pr-gate --run-dir runs/<run_id>

The canonical local run ledger is:

runs/<run_id>/
  task_metadata.json
  final_prompt.md
  model_selection.json
  model_output.md
  validation_report.json
  revision_decision.json
  run_log.jsonl
  metadata.json
  transcript.jsonl
  commands.jsonl
  workspace/
  validation/
  artifacts/

validation_report.json and revision_decision.json are the final acceptance authority. Supporting supervisor reports are local evidence, not a substitute for those Workbench artifacts.

Codex Hooks

Install project-local Codex hooks:

ai-workbench setup codex --project-dir . --task-type code_change

Restart Codex or start a new session, open /hooks, review the project hook, and trust it once. Until a hook event is observed, supervisor status reports Codex coverage as configured but unverified.

Goose MCP

AI Workbench still exposes the same MCP tool lifecycle. Register the server with Goose or another MCP host using:

ai-workbench mcp serve

The seven MCP tools remain:

workbench_open_run
workbench_select_policy_pack
workbench_select_model
workbench_record_execution
workbench_validate_run
workbench_quality_gate
workbench_analyze_runs

PR Gate

Workbench PR acceptance consumes real Workbench run evidence:

ai-workbench pr-gate \
  --run-dir runs/<run_id> \
  --out runs/pr_gate/pr_comment.md \
  --json-out runs/pr_gate/pr_decision.json

Outcomes are exactly:

  • accept

  • needs_review

  • block

Missing, unreadable, or scaffold-only evidence blocks. A green CI run, uploaded artifact, sticky PR comment, or model self-claim is not acceptance evidence.

Bootstrap Assets

To add starter configs, prompts, recipes, docs, and the GitHub PR-gate workflow to a repository:

ai-workbench bootstrap --target .

The bootstrap keeps runs/ ignored.

Package Demo

For a package-only synthetic demo:

ai-workbench demo --target ./workbench-first-run

This shows accept, needs_review, and block PR-gate outcomes with fixture evidence. It is not a real target-repository acceptance run.

Development

python -m pip install -e ".[dev,publish]"
python -m pytest -q -p no:cacheprovider
python -m ruff check . --no-cache
python -m mypy --no-sqlite-cache --no-incremental
ai-workbench demo --target runs/package_demo_smoke
ai-workbench validate --project ai_workbench_mcp --profile scaffold --run-dir runs/scaffold-smoke

Do not commit runs/. Committed sample evidence must be sanitized and live under examples/.

Docs

Recipes:

Sample evidence:

License

Apache-2.0. See LICENSE. MIT-origin attribution for the consolidated Prove It code is retained in NOTICE.

Available Tools

7 tools
workbench_analyze_runsB

Analyze local Workbench run ledgers and write report artifacts.

Inputs: runs_dir, task_type, since, out_dir, +2 more. Returns: object

workbench_open_runC

Create a Workbench run folder and initial evidence artifacts.

Inputs: project (required), task (required), run_dir, prompt, +10 more. Returns: object

workbench_quality_gateD

Run the Workbench quality gate for a run directory.

Inputs: project (required), run_dir (required), mode, risk, +3 more. Returns: object

workbench_record_executionC

Capture Goose/model response text into Workbench evidence artifacts.

Inputs: project (required), run_dir (required), response_text (required), files_touched, +5 more. Returns: object

workbench_select_modelC

Select a Workbench model tier and write model_selection.json.

Inputs: project (required), task_type (required), risk (required), out (required), +10 more. Returns: object

workbench_select_policy_packB

Recommend an advisory Workbench policy pack from task metadata.

Inputs: task_text, task_type, changed_files, prompt, +1 more. Returns: object

workbench_validate_runC

Run deterministic Workbench validation over a run directory.

Inputs: project (required), out_dir (required), profile, changed_files, +2 more. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    C2.9/5.0

    Scored across 7 tools

    Disambiguation5/5

    Each tool has a clearly distinct purpose: analyzing runs, opening runs, quality gating, recording execution, selecting model, selecting policy pack, and validating runs. No overlap in functionality.

    Naming Consistency4/5

    All tools share the 'workbench_' prefix and mostly follow a verb_noun pattern (e.g., analyze_runs, open_run). Minor deviation with 'quality_gate' which is noun_verb, but overall consistent.

    Tool Count5/5

    With 7 tools, the set is well-scoped for an AI workbench run management domain. Not too few to lack utility, not too many to be unwieldy.

    Completeness3/5

    The tools cover core actions like opening, analyzing, recording, and validating runs, but lack lifecycle management tools such as listing, closing, or deleting runs. Some gaps exist for a complete workflow.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      Enables verification of AI coding agent self-reports against git diff truth and a deterministic gate, producing pass/regenerate/reject directives to ensure claimed work matches actual changes.
      6
      144 PyPI
      AGPL 3.0
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables evidence-first regression testing for AI agents by turning production traces into reviewed, replayable cases that gate releases. It supports reproducible, auditable agent evaluation with controlled tool execution, evidence-based judging, and versioned quality gates.
      MIT