# comfyui-test-integrations

Headless, real-workflow integration testing for ComfyUI — queue an actual
workflow file through a real, running ComfyUI instance and get back a
structured pass/fail, without a human at a browser.

Built to close a specific gap: ComfyUI's frontend (LiteGraph, subgraph
blueprints, widget serialization) has real conversion logic between a saved
workflow JSON and the flattened API "prompt" format it actually executes.
Reimplementing that conversion by hand is a losing game — see
`test_workflow_headless.py`'s own docstring for the specific edge case
(`control_after_generate` companion widgets) that broke a from-scratch
attempt. Instead, this drives an actual headless Chromium via Playwright, so
the real frontend does the conversion every time.

## What's here

- **`test_workflow_headless.py`** — the core harness. CLI + importable
  library (`prepare_workflow()`, `run()`). Loads a workflow JSON into a real
  ComfyUI frontend (`window.app.loadGraphData()`), queues it
  (`app.queuePrompt()`), and polls `/history` for the result. Auto-attaches a
  `SaveVideo`/`SaveImage` node onto a bare blueprint's dangling output (the
  upstream Comfy-Org blueprint convention ships without one).
- **`comfyui_mcp_server.py`** — the same logic exposed as a single MCP tool
  (`queue_workflow`), via the official Python `mcp` SDK's FastMCP, so any
  MCP-capable agent (Claude Code, Hermes agent, etc.) can use it without
  knowing Playwright or ComfyUI internals exist.
- **`examples/`** — worked examples, including wiring in
  [Comfy-Org/ComfyUI-test-framework](https://github.com/Comfy-Org/ComfyUI-test-framework)'s
  assertion nodes (`AssertExecuted`, `AssertTensorShape`, ...) for richer
  in-graph checks than "did it execute without error."

## Install

```
pip install -r requirements.txt
python3 -m playwright install chromium   # skip --with-deps if your host
                                          # already has the usual browser libs
```

## CLI usage

```
python3 test_workflow_headless.py <workflow.json> [--url http://host:port] \
    [--prompt "override text"] [--timeout SECONDS]
```

Exit code 0 on success, 1 on any failure (load error, queueing rejected,
execution error, or timeout).

## MCP usage

```
python3 comfyui_mcp_server.py    # stdio server
```

Register with an MCP client, e.g. a project's `.mcp.json` for Claude Code:

```json
{
  "mcpServers": {
    "comfyui-workflow-runner": {
      "command": "python3",
      "args": ["/absolute/path/to/comfyui-test-integrations/comfyui_mcp_server.py"]
    }
  }
}
```

or Hermes agent's `~/.hermes/config.yaml`:

```yaml
mcp_servers:
  comfyui:
    command: python3
    args: ["/absolute/path/to/comfyui-test-integrations/comfyui_mcp_server.py"]
```

Set `COMFYUI_URL` in the environment (or pass `url` per call) to point at a
non-default instance.

## Richer assertions (optional)

For more than "did it execute without an error," install
[Comfy-Org/ComfyUI-test-framework](https://github.com/Comfy-Org/ComfyUI-test-framework)
as a ComfyUI custom node (adds `AssertExecuted`, `AssertTensorShape`,
`AssertInRange`, etc. as regular graph nodes) and wire its assertion nodes
into a copy of the workflow you're testing before queueing it through this
harness. See `examples/build_assertion_test_example.py` for a worked
example — including a real gotcha: **subgraph-internal links use a
different JSON schema than outer-graph links** (a
`{id, origin_id, origin_slot, target_id, target_slot, type}` dict, not the
outer level's flat `[id, from_node, from_slot, to_node, to_slot, type]`
array).

Note `ComfyUI-test-framework`'s own CLI (`comfyci`) is a different,
complementary tool: it expects workflows already in flattened API-prompt
format (baked in via ComfyUI's own save/export, so it can go stale if the
graph is edited without re-exporting) and submits via plain HTTP/WebSocket —
no browser involved. Its `AssertImageMatch` node is explicitly documented as
unsuitable for non-deterministic (e.g. diffusion) output. This repo's harness
is the complement: always-fresh conversion via a real browser, for verifying
"does this generative workflow still run," not pixel-exact regression.

## Wiring into a deploy pipeline

See the `deploy/integration_test.sh` pattern in
[zbrad/ComfyUI](https://github.com/zbrad/ComfyUI)'s `deploy/` directory for
an example of gating a deploy on a real queued workflow, using this repo as
the harness.

## License

MIT — see `LICENSE`.