deskcheck-ai

A free, open-source (MIT), deterministic desktop-test loop for open coding agents. A coding agent (OpenCode, Codex CLI, Claude Code, etc.) calls deskcheck to verify its own GUI/desktop work: launch a named app, perform scripted GUI steps, screenshot, get pass/fail back into its build-fix loop.

It is a test harness, not another agent brain. It decides nothing about what to do — it runs exactly the steps in a YAML test file and reports whether they passed.

Why this exists

Desktop agents that can click anywhere carry an obvious risk: an agent with OS-wide control can touch mail, password managers, and signed-in browsers (Cursor's Computer Use announcement, Oct 23, 2026, published no permission model for exactly this). deskcheck's answer is the headline feature: a named-app allowlist with default deny, enforced at launch and at every input step, with whole categories (password managers, credential stores, system-settings apps, browsers with existing profiles) denied outright. See docs/SECURITY.md.

Related MCP server: clef-use

Quickstart

pip install -e .
# 1. Write a test (steps are data, never code — no shell/exec step exists)
# 2. Run it against the mock driver (hermetic demo, no real desktop needed)
deskcheck run examples/demo-pass.yaml --config examples/config.yaml \
  --driver mock --mock-windows demo-app
# Exit 0 = pass, 1 = fail, 2 = setup/parse error, 3 = interrupted (Ctrl-C)
# Reports: <test-dir>/runs/<name>/report.md + report.html + actions.jsonl
#          + screenshots/<name>.png per screenshot step

On a real Linux desktop, use --driver linux (needs xdotool + scrot); on Windows, --driver windows (PowerShell + .NET, no new dependencies). If the driver is unavailable, deskcheck says so clearly instead of failing mid-run.

One full example test

version: 1
name: demo-pass
app: demo-app
steps:
  - type: launch
    app: demo-app
  - type: wait
    ms: 10
  - type: click
    x: 100
    y: 200
  - type: type
    text: "hello"
  - type: key
    key: Enter
  - type: screenshot
    name: main
  - type: assert
    kind: window-visible
    window: demo-app
  - type: assert
    kind: screenshot-saved
    screenshot: main

Step types, and only these: launch, wait (ms, or window-visible), click (x,y), type (text), key (named key), screenshot (named), assert (window-visible | screenshot-saved | pixel-diff vs a baseline, with tolerance + max-diff-ratio knobs). The parser rejects any unknown step type or unknown key with a clear error, and rejects shell/exec-style keys outright. Timeouts apply to every step; --max-steps caps runaway tests; Ctrl-C stops gracefully and marks the run interrupted.

Pixel-diff baselines (the Playwright-style accept flow)

First run with no baseline fails with a clear "baseline missing" message. Accept the current screenshot as the baseline, then re-run:

deskcheck run examples/pixel-diff.yaml --config examples/config.yaml \
  --driver mock --mock-windows demo-app --update-baselines
deskcheck run examples/pixel-diff.yaml --config examples/config.yaml \
  --driver mock --mock-windows demo-app

On a pixel-diff failure, a red diff image is written next to the report (diff-<name>.png). OCR is optional only: if tesseract is installed you may add your own checks outside deskcheck; it is never in the default path and never required.

MCP setup (for coding agents)

deskcheck speaks MCP over stdio (JSON-RPC 2.0, initialize / tools/list / tools/call / ping). Tools: launch_app, click, type_text, press_key, screenshot, assert_window, run_test (tool failures return isError: true, not a transport error, so the model can read the reason and retry).

{
  "mcpServers": {
    "deskcheck": {
      "command": "deskcheck",
      "args": ["mcp", "--config", "/path/to/config.yaml", "--driver", "linux"]
    }
  }
}

Drop-in agent instructions live in AGENT-SKILL.md: write test → run → read report → fix → re-run.

How deskcheck differs from HQ's Holo4

Holo4 (in the HQ system) is a model-driven autonomous desktop agent: it decides what to do next on the desktop. deskcheck is the opposite side of that loop: it verifies — it runs only the steps a test file spells out, against an allowlist, and returns pass/fail. Interop is documented here only; there is no code coupling to HQ, and deskcheck never modifies HQ files. In a combined workflow, Holo4 (or any agent) builds, deskcheck checks, the agent fixes, deskcheck re-checks.

Docs

• docs/SECURITY.md — permission model + threat model

• docs/RESEARCH.md — what was borrowed from Cursor, Playwright, PyAutoGUI, and MCP conventions

• AGENT-SKILL.md — drop-in instructions for coding agents

Tests

sh test/smoke.sh   # unit + e2e (MockDriver) + CLI + MCP handshake

No network calls in tests. Dependencies: PyYAML only; everything else (including the PNG decoder used for pixel-diff) is stdlib.