ComfyUI Local MCP Server

An MCP server that lets MCP clients (Claude Code, Claude Desktop, …) generate and edit images with local Stable Diffusion models running in ComfyUI. No cloud API or API key is involved.

Forked from tadasant/mcp-server-stability-ai, with the Stability AI cloud client replaced by a ComfyUI client.

Tools

Each result is saved to IMAGE_STORAGE_DIRECTORY and opened in the system image viewer. Matching MCP prompts are provided for each tool.

Models

comfyui-generate-image and comfyui-img2img take a model argument. Its schema describes what each model is for, so the calling LLM picks one from the request. The user can also name one explicitly.

Each model has its own sampler, steps, CFG, CLIP skip and quality tags, defined in src/comfyui/models.ts. To add a model, add an entry there. All of these are SDXL-family checkpoints that load with CheckpointLoaderSimple. Models that need separate text-encoder or VAE loaders (for example Z-Image or Flux) need their own workflow.

The Hugging Face sources are RealVisXL V5.0, Pony Diffusion V6 XL, Illustrious XL v0.1 and SDXL base 1.0. The upscaler is RealESRGAN_x4plus.pth, placed in the upscale_models folder.

How it works

Each tool fills in a workflow template from src/workflows/. The templates are in ComfyUI's API format, with readable node ids such as sampler and positive. The server then:

1. uploads any input image with POST /upload/image

2. submits the workflow with POST /prompt

3. polls GET /history/{prompt_id} until the run completes or fails

4. fetches the output image with GET /view

The client is src/comfyui/comfyUiClient.ts. ComfyUI keeps its own copies of inputs and outputs under input/mcp/ and output/mcp/.

Setup

Related MCP server: ComfyUI MCP Server

1. Backend

Install SwarmUI and let its installer set up the ComfyUI backend. SwarmUI's Generate tab and ComfyUI node editor are useful for trying things by hand. Then put the models above into SwarmUI's Models/Stable-Diffusion/ and Models/upscale_models/ folders.

When SwarmUI starts ComfyUI, ComfyUI listens on port 7821. A standalone ComfyUI uses 8188. The background-removal tool uses SwarmUI's built-in SwarmRemBg node, so it needs ComfyUI started by SwarmUI. On first use, rembg downloads its u2net model.

Transparent output (optional): transparent: true uses ComfyUI-layerdiffuse. Clone it into ComfyUI's custom_nodes/ folder (with SwarmUI, dlbackend/ComfyUI/custom_nodes/) and restart SwarmUI. Its model files download automatically on first use. Its LayeredDiffusionDecodeRGBA node doesn't work with current ComfyUI, so the server builds the RGBA output from LayeredDiffusionDecode, InvertMask and JoinImageWithAlpha instead (see src/comfyui/layerDiffuse.ts). Transparency works best with sdxl and realvis.

SwarmUI has to be running whenever the tools are used. Otherwise they fail with Cannot reach ComfyUI at ….

2. Build

npm install
npm run build   # compiles to build/ and copies src/workflows there

Rebuild after any change. MCP clients run build/index.js.

3. Register with an MCP client

Claude Code:

claude mcp add --scope user comfyui-local \
  -e COMFYUI_URL=http://127.0.0.1:7821 \
  -e IMAGE_STORAGE_DIRECTORY=$HOME/Pictures/comfyui-mcp \
  -- "$(command -v node)" /path/to/this/repo/build/index.js

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "comfyui-local": {
      "command": "node",
      "args": ["/path/to/this/repo/build/index.js"],
      "env": {
        "COMFYUI_URL": "http://127.0.0.1:7821",
        "IMAGE_STORAGE_DIRECTORY": "/path/to/images"
      }
    }
  }
}

IMAGE_STORAGE_DIRECTORY must already exist.

Environment variables

SSE mode

node build/index.js --sse serves over HTTP on port 3020 and stores images in Google Cloud Storage instead of the local filesystem. This mode is inherited from upstream and hasn't been tested with the ComfyUI backend. It requires GCS_PROJECT_ID, GCS_CLIENT_EMAIL, GCS_PRIVATE_KEY and GCS_BUCKET_NAME. Its multi-tenancy is naive: images are split by requester IP and are publicly readable.

Making game UI assets

What we found while building the UI tools, with SDXL-family models:

• Text-only prompts don't work well for UI. The models rarely saw clean, empty UI frames, so they fill panels with fake text, draw whole UI screens, or ignore "no humans". Illustrious/Pony tags like dialogue box make it worse, because on the image boards those tags mark screenshots full of text.

• Draw the layout, then let the model render it. Use comfyui-ui-frame for frames and panels. For other shapes, draw a rough version and use comfyui-controlnet (canny, denoise 0.8–0.9). The drawing fixes the layout, and the model adds the material and ornament.

• Draw in the target colors. At denoise below 1 the draft's colors carry through, so a navy draft won't turn into parchment.

• Protect empty areas with a mask. comfyui-ui-frame repaints the panel at only panelDetail (default 0.3), so it can't grow objects or text, whatever the seed.

• Transparency: for UI frames, the exact geometry cut-out gives crisp edges and full color. transparent: true on generate-image/controlnet (LayerDiffuse) is best for free-form items like icons and potions. Starting LayerDiffuse from a drawn layout (denoise < 1) gave washed-out colors.

• Use sdxl for UI. It gave the cleanest frames. Negative prompt: (text:1.4), (letters:1.3), watermark, people, face, (emblem in the middle:1.3), user interface elements, buttons, icons, perspective, blurry.

• Thin frames with lush decoration: use overhang with a small borderWidth. For example, a 28 px wooden bar with overhang: 48 lets ivy drape over the panel edge. Without it, decoration can only grow inside the border strip.

• Organic frames (frames made of vines, branches, bones…): comfyui-ui-frame doesn't suit these, because its solid border band makes the model paint a flat plank. Instead, draw a guide of several wavy, crossing strands along a rectangle, with leaf shapes at the corners, on a black background. Then run comfyui-controlnet with mode: canny, strength: 0.6, denoise: 1, transparent: true and model: sdxl. Plain text-to-image draws the right style but a random shape (ovals, single corners). Plain wavy lines come out as braided rope, so vary the strand width and waviness.

Capabilities and limits

What we found testing a range of subjects on this setup (SDXL-family models, 8 GB GPU):

Rule of thumb: show the model instead of telling it. A drawn shape, a reference image or an existing composition is far more reliable than a longer prompt.

Practical workarounds:

• Hybrid creatures: first generate a composition that works, even if the colors are wrong, then fix the details with comfyui-img2img at denoise 0.5–0.6. The winged Holstein worked this way after text-only attempts failed. Avoid words that pull toward another animal ("pegasus" becomes a horse, "eagle wings" becomes an eagle).

• Colors bleeding between parts: "white wings" also turned the cow white. Give each part its own color and weight the important one, e.g. (black and white spotted Holstein cow:1.4).

• Faces: state age, ethnicity and grooming explicitly when matching a reference (e.g. "young, clean-shaven"). The realistic model otherwise tends toward a rugged, bearded adult.

• img2img reinterprets vague shapes: at denoise 0.5 a lumpy rock became a wolf and then a camel statue. Name the background ("large rock outcrop") and add the unwanted thing to the negative prompt.

Not installed yet, and what each would fix:

• IP-Adapter: use a reference image for appearance (costume, face) instead of describing it. This fixes the lost costume details.

• OpenPose ControlNet: copy a pose from any photo. The Union ControlNet already supports it, but it needs a pose-detector extension (e.g. comfyui_controlnet_aux).

• Inpainting: repaint just one region, e.g. add a missing eyepatch.

Testing

npm test                   # ~3 s, no GPU: templates, model settings, client and MCP server vs. a fake ComfyUI
npm run test:integration   # also runs against the real ComfyUI (SwarmUI running, GPU free); several minutes

• npm test checks that the workflow templates are consistent, and that each model's settings and prompt tags are applied. It runs the ComfyUI client against a fake ComfyUI HTTP server (success, execution errors, validation errors, timeouts, unreachable server), then starts build/index.js and calls every tool over MCP.

• npm run test:integration also checks every template against the live server's /object_info (node types, inputs, checkpoints, samplers, upscale model). It then generates an image with each model and runs img2img, upscale and remove-background, checking the output sizes and transparency.

Tests use Node's built-in test runner and live in test/. They run against build/, and both scripts build first.

Roadmap (Phase 2)

• Inpaint / outpaint

• ControlNet tools (structure, sketch, style)

• Relight, search-and-recolor

• Z-Image Turbo support (separate text-encoder, VAE and model-sampling workflow)

See sd-mcp-fork-spec.md for the original plan.