# comfyui-mcp

A TypeScript [MCP](https://modelcontextprotocol.io) server that lets an AI agent drive a remote [ComfyUI](https://github.com/comfyanonymous/ComfyUI) instance end-to-end — queue workflows, iterate on prompts and seeds, watch progress, view outputs as inline images, curate winners, download models, and author whole node graphs as Python — without ever touching the web canvas.

Built for and battle-tested on an NVIDIA DGX Spark (GB10, unified memory) that doubles as an LLM-serving box, which shaped some of the more interesting parts of the design (see [GPU-yield handshake](#gpu-yield-handshake) below).

## Why

ComfyUI's node canvas is great for humans and terrible for agents. The HTTP API alone isn't enough either: workflow files live on the GPU host, outputs land on the GPU host, and model downloads need to happen on the GPU host. This server gives an agent a complete operational surface over both channels:

- **HTTP** to the ComfyUI API for queueing, progress, history, and image retrieval
- **SSH** to the GPU host for workflow file r/w, output listing, model downloads (via remote `aria2c`), and ComfyScript execution

## Tool surface (16 tools)

| Group | Tools |
|---|---|
| Stats | `system_stats` (RAM/VRAM, version, queue depth) |
| Viewing | `list_outputs`, `view_image`, `view_latest` — images come back as MCP image blocks |
| Workflows | `list_workflows`, `read_workflow`, `write_workflow` |
| Queueing | `queue_workflow` (with overrides), `get_progress`, `cancel_queue` |
| Waiting | `wait_for_image` — blocks until the render lands |
| Batching | `queue_batch` — up to 8 prompt/seed/param variants in one call |
| Models | `list_models`, `download_model` (Civitai + HuggingFace, token-aware) |
| Curation | `pick_top` — copy winners to a named folder remotely + scp them local |
| Scripting | `run_comfyscript` — author the graph as Python |

## The override engine

`queue_workflow` takes a saved workflow as a base and applies structured overrides — `prompt`, `negative`, `checkpoint`, `seed`, `steps`, `cfg`, `guidance`, `sampler`, `scheduler`, `width`, `height`, `loras` — by rewriting the graph before submission. The interesting cases:

- **LoRA injection is architecture-aware**: FLUX models load via `UNETLoader` and take CLIP from a separate `DualCLIPLoader`, so injected LoRAs use `LoraLoaderModelOnly` with no CLIP rewire; SDXL checkpoints get the classic `LoraLoader` treatment. The engine detects which world it's in.
- **Unknown targets fail loudly**: overriding a field the workflow has no node for returns `override_target_not_found:<field>` instead of silently generating the wrong thing.

Canonical starter templates for six model families (FLUX, FLUX.2, SDXL, Illustrious/anime, Qwen-Image, Z-Image Turbo) ship in `templates/`, generated deterministically from known-good source workflows by `scripts/make_templates.py` (strips LoRAs, normalizes titles, randomizes seeds, converts fixed-size upscales to relative ones so width/height overrides stay aspect-correct).

## run_comfyscript — the escape hatch

JSON workflow + overrides covers routine text-to-image iteration. It does not cover a two-stage upscale pipeline, a video model with audio conditioning, or anything you'd actually need the canvas for. `run_comfyscript` accepts a Python snippet using [ComfyScript](https://github.com/Chaoses-Ib/ComfyScript), wraps it in the runtime preamble, executes it in the ComfyUI venv on the GPU host over SSH, and reports back only the *newly created* output files. Every node installed on the server (~2900 including custom nodes) is callable as a Python function, so the agent can author arbitrary graphs as code.

## GPU-yield handshake

On a unified-memory box the image side and the LLM side compete for the same physical RAM, and a 35GB diffusion model colliding with a resident LLM is an OOM. Before any GPU work, `src/yield.ts` asks the LLM router ([llama-swap](https://github.com/mostlygeek/llama-swap)) to unload its models — called from the shared `queuePrompt()` path so every current and future queue tool inherits it, and separately from `run_comfyscript` which bypasses that path. It's best-effort by design: if the router is down or slow, the server logs and proceeds rather than blocking a generation. The mirror-image policy (LLM loads freeing an *idle* ComfyUI) lives host-side.

## Setup

```bash
npm install && npm run build
```

Register with your MCP client (Claude Code shown), pointing the env at your ComfyUI host:

```jsonc
// ~/.claude.json → mcpServers
"comfyui": {
  "command": "node",
  "args": ["/path/to/comfyui-mcp/dist/index.js"],
  "env": {
    "COMFYUI_HOST": "http://<gpu-host>:8188",
    "SPARK_SSH_ALIAS": "<ssh-alias>",            // from ~/.ssh/config
    "SPARK_COMFYUI_ROOT": "/path/to/ComfyUI",    // on the GPU host
    "SPARK_COMFYUI_PYTHON": "/path/to/venv/bin/python",
    "LLAMASWAP_HOST": "http://<gpu-host>:8089",  // optional, GPU-yield
    "CIVITAI_API_TOKEN": "${CIVITAI_API_TOKEN}", // optional, gated downloads
    "HF_TOKEN": "${HF_TOKEN}"
  }
}
```

SSH access to the GPU host (key-based, via the alias) is required for the workflow/model/comfyscript tools; the queue/view tools work over HTTP alone.

## Testing

```bash
npm test          # 81 tests, fully offline
npm run smoke -- --live   # hits the real GPU host
```

Every tool handler is a pure function taking injectable deps (`handleX(args, deps)`), so the entire suite runs offline against fakes — no ComfyUI instance needed. The live smoke script is separate and explicit.

## Docs

- [`docs/realism-findings.md`](docs/realism-findings.md) — empirical findings on photorealistic portrait generation and character identity consistency with FLUX.1 Krea: what actually makes a generated image read as a modern photo, which prompt signals break identity across a set, and documented demographic biases in scene priors.

## License

MIT