# ComfyUI Local MCP Server

An [MCP](https://modelcontextprotocol.io/) server that lets MCP clients (Claude Code, Claude Desktop, …) generate and edit images with **local Stable Diffusion models** running in [ComfyUI](https://github.com/comfyanonymous/ComfyUI). No cloud API or API key is involved.

Forked from [tadasant/mcp-server-stability-ai](https://github.com/tadasant/mcp-server-stability-ai), with the Stability AI cloud client replaced by a ComfyUI client.

# Tools

| Tool | What it does |
|---|---|
| `comfyui-generate-image` | Text to image. Takes a prompt, negative prompt, aspect ratio, seed, `model` and `transparent`. |
| `comfyui-img2img` | Reworks an existing image from a prompt. `denoise` (0.05–1) sets how far it may drift from the original. |
| `comfyui-controlnet` | Follows a guide image's structure (ControlNet Union ProMax). `canny` mode keeps its outlines and takes materials from the prompt. `tile` keeps its layout and colors. Good for turning a drawn shape into a game asset. Also supports `transparent`. |
| `comfyui-ui-frame` | Game UI frame or panel (dialogue box, window, tooltip) ready for 9-slice. Draws the layout in the given colors, repaints the border fully and the panel only lightly (Differential Diffusion), cuts out the background, and writes `<name>.9slice.json` with Unity and Godot margins. `overhang` lets decoration like vines spill over the panel edge, for a thin but lush border. |
| `comfyui-upscale` | 2x or 4x upscale with an ESRGAN model. Content is unchanged. |
| `comfyui-remove-background` | Transparent-background PNG, via rembg. |
| `comfyui-0-list-resources` | Lists images in the storage directory, so they can be used as inputs. |

Each result is saved to `IMAGE_STORAGE_DIRECTORY` and opened in the system image viewer. Matching MCP prompts are provided for each tool.

## Models

`comfyui-generate-image` and `comfyui-img2img` take a `model` argument. Its schema describes what each model is for, so the calling LLM picks one from the request. The user can also name one explicitly.

| `model` | Checkpoint (relative to the ComfyUI checkpoints folder) | Use for |
|---|---|---|
| `realvis` (default) | `RealVisXL/RealVisXL_V5.0_fp16.safetensors` | Photorealistic images |
| `pony` | `Pony/ponyDiffusionV6XL_v6StartWithThisOne.safetensors` | Illustrated, anime and cartoon art |
| `illustrious` | `Illustrious/Illustrious-XL-v0.1.safetensors` | Anime and character art |
| `sdxl` | `OfficialStableDiffusion/sd_xl_base_1.0.safetensors` | General and artistic styles |

Each model has its own sampler, steps, CFG, CLIP skip and quality tags, defined in [`src/comfyui/models.ts`](src/comfyui/models.ts). To add a model, add an entry there. All of these are SDXL-family checkpoints that load with `CheckpointLoaderSimple`. Models that need separate text-encoder or VAE loaders (for example Z-Image or Flux) need their own workflow.

The Hugging Face sources are [RealVisXL V5.0](https://huggingface.co/SG161222/RealVisXL_V5.0), [Pony Diffusion V6 XL](https://huggingface.co/LyliaEngine/Pony_Diffusion_V6_XL), [Illustrious XL v0.1](https://huggingface.co/OnomaAIResearch/Illustrious-xl-early-release-v0) and [SDXL base 1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0). The upscaler is [RealESRGAN_x4plus.pth](https://github.com/xinntao/Real-ESRGAN/releases/tag/v0.1.0), placed in the `upscale_models` folder.

# How it works

Each tool fills in a workflow template from [`src/workflows/`](src/workflows). The templates are in ComfyUI's API format, with readable node ids such as `sampler` and `positive`. The server then:

1. uploads any input image with `POST /upload/image`
2. submits the workflow with `POST /prompt`
3. polls `GET /history/{prompt_id}` until the run completes or fails
4. fetches the output image with `GET /view`

The client is [`src/comfyui/comfyUiClient.ts`](src/comfyui/comfyUiClient.ts). ComfyUI keeps its own copies of inputs and outputs under `input/mcp/` and `output/mcp/`.

# Setup

## 1. Backend

Install [SwarmUI](https://github.com/mcmonkeyprojects/SwarmUI) and let its installer set up the ComfyUI backend. SwarmUI's Generate tab and ComfyUI node editor are useful for trying things by hand. Then put the models above into SwarmUI's `Models/Stable-Diffusion/` and `Models/upscale_models/` folders.

When SwarmUI starts ComfyUI, ComfyUI listens on **port 7821**. A standalone ComfyUI uses 8188. The background-removal tool uses SwarmUI's built-in `SwarmRemBg` node, so it needs ComfyUI started by SwarmUI. On first use, rembg downloads its u2net model.

**Transparent output (optional):** `transparent: true` uses [ComfyUI-layerdiffuse](https://github.com/huchenlei/ComfyUI-layerdiffuse). Clone it into ComfyUI's `custom_nodes/` folder (with SwarmUI, `dlbackend/ComfyUI/custom_nodes/`) and restart SwarmUI. Its model files download automatically on first use. Its `LayeredDiffusionDecodeRGBA` node doesn't work with current ComfyUI, so the server builds the RGBA output from `LayeredDiffusionDecode`, `InvertMask` and `JoinImageWithAlpha` instead (see [`src/comfyui/layerDiffuse.ts`](src/comfyui/layerDiffuse.ts)). Transparency works best with `sdxl` and `realvis`.

SwarmUI has to be running whenever the tools are used. Otherwise they fail with `Cannot reach ComfyUI at …`.

## 2. Build

```bash
npm install
npm run build   # compiles to build/ and copies src/workflows there
```

Rebuild after any change. MCP clients run `build/index.js`.

## 3. Register with an MCP client

Claude Code:

```bash
claude mcp add --scope user comfyui-local \
  -e COMFYUI_URL=http://127.0.0.1:7821 \
  -e IMAGE_STORAGE_DIRECTORY=$HOME/Pictures/comfyui-mcp \
  -- "$(command -v node)" /path/to/this/repo/build/index.js
```

Claude Desktop (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "comfyui-local": {
      "command": "node",
      "args": ["/path/to/this/repo/build/index.js"],
      "env": {
        "COMFYUI_URL": "http://127.0.0.1:7821",
        "IMAGE_STORAGE_DIRECTORY": "/path/to/images"
      }
    }
  }
}
```

`IMAGE_STORAGE_DIRECTORY` must already exist.

## Environment variables

| Variable | Default | Description |
|---|---|---|
| `COMFYUI_URL` | `http://127.0.0.1:7821` | ComfyUI server address |
| `COMFYUI_DEFAULT_MODEL` | `realvis` | Model used when a call doesn't pass `model` |
| `COMFYUI_CONTROLNET_MODEL` | `xinsir-union-sdxl-promax.safetensors` | ControlNet Union file in ComfyUI's `controlnet` folder ([source](https://huggingface.co/xinsir/controlnet-union-sdxl-1.0), file `diffusion_pytorch_model_promax.safetensors`) |
| `COMFYUI_UPSCALE_MODEL` | `RealESRGAN_x4plus.pth` | File in ComfyUI's `upscale_models` folder. Must be a 4x model. |
| `IMAGE_STORAGE_DIRECTORY` | `/tmp/mcp-server-comfyui-local` (Windows: `C:\Windows\Temp\mcp-server-comfyui-local`) | Where results are saved |
| `SAVE_METADATA` | `true` | Write a `.txt` next to each image with the request parameters (model, seed, …) |
| `SAVE_METADATA_FAILED` | `true` | Write `<name>-failed-<timestamp>.txt` for failed requests, including the error |
| `OPEN_IMAGES` | `true` | Open each result in the system image viewer. Set `false` to disable. |

## SSE mode

`node build/index.js --sse` serves over HTTP on port 3020 and stores images in Google Cloud Storage instead of the local filesystem. This mode is inherited from upstream and hasn't been tested with the ComfyUI backend. It requires `GCS_PROJECT_ID`, `GCS_CLIENT_EMAIL`, `GCS_PRIVATE_KEY` and `GCS_BUCKET_NAME`. Its multi-tenancy is naive: images are split by requester IP and are publicly readable.

# Making game UI assets

What we found while building the UI tools, with SDXL-family models:

- **Text-only prompts don't work well for UI.** The models rarely saw clean, empty UI frames, so they fill panels with fake text, draw whole UI screens, or ignore "no humans". Illustrious/Pony tags like `dialogue box` make it worse, because on the image boards those tags mark screenshots full of text.
- **Draw the layout, then let the model render it.** Use `comfyui-ui-frame` for frames and panels. For other shapes, draw a rough version and use `comfyui-controlnet` (`canny`, denoise 0.8–0.9). The drawing fixes the layout, and the model adds the material and ornament.
- **Draw in the target colors.** At denoise below 1 the draft's colors carry through, so a navy draft won't turn into parchment.
- **Protect empty areas with a mask.** `comfyui-ui-frame` repaints the panel at only `panelDetail` (default 0.3), so it can't grow objects or text, whatever the seed.
- **Transparency:** for UI frames, the exact geometry cut-out gives crisp edges and full color. `transparent: true` on generate-image/controlnet (LayerDiffuse) is best for free-form items like icons and potions. Starting LayerDiffuse from a drawn layout (denoise < 1) gave washed-out colors.
- **Use `sdxl` for UI.** It gave the cleanest frames. Negative prompt: `(text:1.4), (letters:1.3), watermark, people, face, (emblem in the middle:1.3), user interface elements, buttons, icons, perspective, blurry`.
- **Thin frames with lush decoration:** use `overhang` with a small `borderWidth`. For example, a 28 px wooden bar with `overhang: 48` lets ivy drape over the panel edge. Without it, decoration can only grow inside the border strip.
- **Organic frames** (frames made of vines, branches, bones…): `comfyui-ui-frame` doesn't suit these, because its solid border band makes the model paint a flat plank. Instead, draw a guide of several wavy, crossing strands along a rectangle, with leaf shapes at the corners, on a black background. Then run `comfyui-controlnet` with `mode: canny`, `strength: 0.6`, `denoise: 1`, `transparent: true` and `model: sdxl`. Plain text-to-image draws the right style but a random shape (ovals, single corners). Plain wavy lines come out as braided rope, so vary the strand width and waviness.

# Capabilities and limits

What we found testing a range of subjects on this setup (SDXL-family models, 8 GB GPU):

| Works well | Struggles |
|---|---|
| Single common subjects in any style (photo, anime, painterly) | Long, detailed descriptions: SDXL's text encoder reads about 75 tokens (50–60 words) at a time, so with a dozen specific details the later ones get dropped (an eyepatch, a fur tail) |
| Restyling an existing image with img2img (a game screenshot made photorealistic, a scene turned into stained glass) | Invented creatures: "cow with eagle wings" drew an eagle standing on a cow, and "Holstein cow as a pegasus" drew a horse |
| Following a supplied shape with ControlNet (frames, panels, vine borders) | Changing a pose while keeping an exact costume or face: text can't carry the whole look, and img2img can't change the pose |
| Upscaling, background removal, transparent icons | Empty UI elements from text alone (they fill with fake text or clutter) |
| | Exact placement ("wings from the shoulders", "eyepatch on the left eye") is a hint, not a guarantee |

**Rule of thumb:** show the model instead of telling it. A drawn shape, a reference image or an existing composition is far more reliable than a longer prompt.

Practical workarounds:
- **Hybrid creatures:** first generate a composition that works, even if the colors are wrong, then fix the details with `comfyui-img2img` at denoise 0.5–0.6. The winged Holstein worked this way after text-only attempts failed. Avoid words that pull toward another animal ("pegasus" becomes a horse, "eagle wings" becomes an eagle).
- **Colors bleeding between parts:** "white wings" also turned the cow white. Give each part its own color and weight the important one, e.g. `(black and white spotted Holstein cow:1.4)`.
- **Faces:** state age, ethnicity and grooming explicitly when matching a reference (e.g. "young, clean-shaven"). The realistic model otherwise tends toward a rugged, bearded adult.
- **img2img reinterprets vague shapes:** at denoise 0.5 a lumpy rock became a wolf and then a camel statue. Name the background ("large rock outcrop") and add the unwanted thing to the negative prompt.

Not installed yet, and what each would fix:
- **IP-Adapter:** use a reference image for appearance (costume, face) instead of describing it. This fixes the lost costume details.
- **OpenPose ControlNet:** copy a pose from any photo. The Union ControlNet already supports it, but it needs a pose-detector extension (e.g. `comfyui_controlnet_aux`).
- **Inpainting:** repaint just one region, e.g. add a missing eyepatch.

# Testing

```bash
npm test                   # ~3 s, no GPU: templates, model settings, client and MCP server vs. a fake ComfyUI
npm run test:integration   # also runs against the real ComfyUI (SwarmUI running, GPU free); several minutes
```

- `npm test` checks that the workflow templates are consistent, and that each model's settings and prompt tags are applied. It runs the ComfyUI client against a fake ComfyUI HTTP server (success, execution errors, validation errors, timeouts, unreachable server), then starts `build/index.js` and calls every tool over MCP.
- `npm run test:integration` also checks every template against the live server's `/object_info` (node types, inputs, checkpoints, samplers, upscale model). It then generates an image with each model and runs img2img, upscale and remove-background, checking the output sizes and transparency.

Tests use Node's built-in test runner and live in [`test/`](test). They run against `build/`, and both scripts build first.

# Roadmap (Phase 2)

- Inpaint / outpaint
- ControlNet tools (structure, sketch, style)
- Relight, search-and-recolor
- Z-Image Turbo support (separate text-encoder, VAE and model-sampling workflow)

See [`sd-mcp-fork-spec.md`](sd-mcp-fork-spec.md) for the original plan.