comfyui-test-integrations

Headless, real-workflow integration testing for ComfyUI — queue an actual workflow file through a real, running ComfyUI instance and get back a structured pass/fail, without a human at a browser.

Built to close a specific gap: ComfyUI's frontend (LiteGraph, subgraph blueprints, widget serialization) has real conversion logic between a saved workflow JSON and the flattened API "prompt" format it actually executes. Reimplementing that conversion by hand is a losing game — see test_workflow_headless.py's own docstring for the specific edge case (control_after_generate companion widgets) that broke a from-scratch attempt. Instead, this drives an actual headless Chromium via Playwright, so the real frontend does the conversion every time.

What's here

• test_workflow_headless.py — the core harness. CLI + importable library (prepare_workflow(), run()). Loads a workflow JSON into a real ComfyUI frontend (window.app.loadGraphData()), queues it (app.queuePrompt()), and polls /history for the result. Auto-attaches a SaveVideo/SaveImage node onto a bare blueprint's dangling output (the upstream Comfy-Org blueprint convention ships without one).

• comfyui_mcp_server.py — the same logic exposed as a single MCP tool (queue_workflow), via the official Python mcp SDK's FastMCP, so any MCP-capable agent (Claude Code, Hermes agent, etc.) can use it without knowing Playwright or ComfyUI internals exist.

• examples/ — worked examples, including wiring in Comfy-Org/ComfyUI-test-framework's assertion nodes (AssertExecuted, AssertTensorShape, ...) for richer in-graph checks than "did it execute without error."

Related MCP server: Unity MCP Test Loop

Install

pip install -r requirements.txt
python3 -m playwright install chromium   # skip --with-deps if your host
                                          # already has the usual browser libs

CLI usage

python3 test_workflow_headless.py <workflow.json> [--url http://host:port] \
    [--prompt "override text"] [--timeout SECONDS]

Exit code 0 on success, 1 on any failure (load error, queueing rejected, execution error, or timeout).

MCP usage

python3 comfyui_mcp_server.py    # stdio server

Register with an MCP client, e.g. a project's .mcp.json for Claude Code:

{
  "mcpServers": {
    "comfyui-workflow-runner": {
      "command": "python3",
      "args": ["/absolute/path/to/comfyui-test-integrations/comfyui_mcp_server.py"]
    }
  }
}

or Hermes agent's ~/.hermes/config.yaml:

mcp_servers:
  comfyui:
    command: python3
    args: ["/absolute/path/to/comfyui-test-integrations/comfyui_mcp_server.py"]

Set COMFYUI_URL in the environment (or pass url per call) to point at a non-default instance.

Richer assertions (optional)

For more than "did it execute without an error," install Comfy-Org/ComfyUI-test-framework as a ComfyUI custom node (adds AssertExecuted, AssertTensorShape, AssertInRange, etc. as regular graph nodes) and wire its assertion nodes into a copy of the workflow you're testing before queueing it through this harness. See examples/build_assertion_test_example.py for a worked example — including a real gotcha: subgraph-internal links use a different JSON schema than outer-graph links (a {id, origin_id, origin_slot, target_id, target_slot, type} dict, not the outer level's flat [id, from_node, from_slot, to_node, to_slot, type] array).

Note ComfyUI-test-framework's own CLI (comfyci) is a different, complementary tool: it expects workflows already in flattened API-prompt format (baked in via ComfyUI's own save/export, so it can go stale if the graph is edited without re-exporting) and submits via plain HTTP/WebSocket — no browser involved. Its AssertImageMatch node is explicitly documented as unsuitable for non-deterministic (e.g. diffusion) output. This repo's harness is the complement: always-fresh conversion via a real browser, for verifying "does this generative workflow still run," not pixel-exact regression.

Wiring into a deploy pipeline

See the deploy/integration_test.sh pattern in zbrad/ComfyUI's deploy/ directory for an example of gating a deploy on a real queued workflow, using this repo as the harness.

License

MIT — see LICENSE.