ComfyUI MCP Server

# Implementation Status & Next Steps This document captures the current progress of the ComfyUI MCP server relative to the roadmap in [`docs/server-plan.md`](./server-plan.md). It highlights what has shipped, enumerates outstanding work, and recommends the next areas of focus. ## Completed Milestones - **Project scaffolding (Plan §1.1)**: Python package lives under `src/comfyui_mcp/` with cohesive modules for configuration, asset discovery, workflow parsing, mutation helpers, HTTP client, server façade, and CLI entry point.【F:src/comfyui_mcp/__init__.py†L1-L14】【F:src/comfyui_mcp/server.py†L18-L82】【F:src/comfyui_mcp/cli.py†L1-L37】 - **Configuration loader foundation (Plan §1.2)**: `ComfyUISettings`/`load_settings` support TOML files, environment overrides, feature toggles, parameter bounds, and asset directory placeholders.【F:src/comfyui_mcp/config.py†L13-L195】 - **Workflow discovery & baseline semantics (Plan §1.3, §2.1-§2.4)**: `WorkflowDiscovery` loads JSON templates, validates structure, and builds semantic summaries that recognise prompts, checkpoint/LoRA/VAE loaders, samplers, text encoders, resolution controllers, and output nodes.【F:src/comfyui_mcp/workflow.py†L68-L210】 - **Mutation primitives (Plan §3.1)**: `WorkflowMutator` can edit prompts, checkpoints, VAE selections, LoRA assignments (including strength values), sampler CFG/steps/seed, and resolution while tracking diffs.【F:src/comfyui_mcp/mutations.py†L28-L148】 - **Execution façade (Plan §4.1)**: `ComfyUIClient` submits workflows, polls queue/history endpoints, fetches images, and can stream updates via polling callbacks; `ComfyUIMCPServer` wraps list/describe/customize/execute helpers with range validation and asset checks.【F:src/comfyui_mcp/client.py†L15-L73】【F:src/comfyui_mcp/server.py†L27-L188】 - **Basic tooling & examples (Plan §5.1 & §9.3)**: CLI exposes `list`, `describe`, and `assets`; sample workflow and README give minimal usage guidance.【F:src/comfyui_mcp/cli.py†L11-L36】【F:workflows/sample_basic.json†L1-L40】【F:README.md†L1-L24】 - **Initial asset management tests (Plan §6.1/§8.1)**: `AssetCatalog` scans configured directories and enforces safe names, with pytest coverage verifying discovery and validation.【F:src/comfyui_mcp/assets.py†L33-L93】【F:tests/test_assets.py†L9-L56】 - **Workflow & asset refresh controls (Plan §1.2/§6.1)**: Runtime overrides, directory validation, and opt-in workflow watching ensure templates and assets stay in sync without restarts.【F:src/comfyui_mcp/server.py†L18-L188】【F:src/comfyui_mcp/cli.py†L11-L45】 - **FastMCP runtime & manifest (Plan §5.1/§9.1)**: A FastMCP-powered runtime plus `mcp.json` manifest make third-party IDE integration straightforward.【F:src/comfyui_mcp/runtime.py†L1-L133】【F:mcp.json†L1-L12】 ## Outstanding Work by Plan Section ### §1 – Project Scaffolding & Configuration - Implement efficient filesystem watching or cache invalidation so large template directories do not require full rescans on every request when refresh mode is enabled.【F:src/comfyui_mcp/server.py†L27-L188】 ### §2 – Workflow Introspection & Node Interpretation - Expand semantic classification to cover additional node families (conditioning combiners, refiner stages, ControlNet, etc.) and handle multi-stage/branching workflows more accurately than the current keyword heuristics.【F:src/comfyui_mcp/workflow.py†L172-L295】 - Introduce an extensible registry/normalisation layer for vendor-specific or custom node names beyond simple keyword matching.【F:src/comfyui_mcp/workflow.py†L239-L295】 - Improve prompt-role inference for workflows lacking conventional slot ordering and capture stage-level context (e.g., per sampler or branch) when building summaries.【F:src/comfyui_mcp/workflow.py†L212-L233】 ### §3 – Template Mutation API - Support text encoder/model swaps, scheduler selection, batch sweeps, seed strategies, and explicit stage targeting when multiple samplers are present (current mutator edits all samplers uniformly).【F:src/comfyui_mcp/mutations.py†L47-L148】 - Enforce graph integrity during mutations (relink dependent nodes, insert/remove auxiliary LoRA nodes, validate compatible assets) instead of assuming existing topology remains valid.【F:src/comfyui_mcp/mutations.py†L80-L148】 - Produce richer diff summaries (grouped by stage/component) and accept multiple mutation operations per request payload, including combinations of prompts, assets, and sampler tweaks.【F:src/comfyui_mcp/mutations.py†L28-L148】 ### §4 – Execution Pipeline - Add WebSocket support (or smarter polling with backoff) for real-time streaming where ComfyUI supports it; current implementation only polls REST endpoints.【F:src/comfyui_mcp/client.py†L26-L59】 - Collect execution outputs with metadata (preview vs saved paths, stage attribution) and surface convenient download handles in `execute_workflow` responses.【F:src/comfyui_mcp/server.py†L53-L81】 - Implement batch execution flows that submit parameter variations, track prompt IDs per variation, and collate results for multi-run requests.【F:src/comfyui_mcp/server.py†L53-L81】 - Harden error handling with retries, graceful degradation for missing assets, configurable timeouts, and clear failure messaging during submission, streaming, and downloads.【F:src/comfyui_mcp/client.py†L26-L59】【F:src/comfyui_mcp/server.py†L53-L188】 ### §5 – LLM-Facing Abstractions - Define JSON schemas/enums for each MCP tool request & response and integrate with the official MCP tooling stack (e.g., register tools via the `mcp` library) rather than exposing only ad-hoc async methods/CLI.【F:src/comfyui_mcp/server.py†L27-L188】【F:src/comfyui_mcp/cli.py†L11-L36】 - Enrich responses with conversational hints, change logs, and asset references suitable for downstream agents once schemas are formalised.【F:src/comfyui_mcp/server.py†L43-L177】 ### §6 – Model & Asset Management - Expand asset metadata beyond filenames (e.g., model type, file size, compatibility tags) and support configurable fuzzy matching behaviours.【F:src/comfyui_mcp/assets.py†L33-L93】 - Provide richer metadata and incremental refresh APIs beyond the current on-demand rescan to expose new assets with additional context.【F:src/comfyui_mcp/assets.py†L33-L93】 ### §7 – Safety, Robustness & Observability - Apply parameter bounds and text sanitisation across all mutation paths (e.g., prompts, LoRA strengths) and ensure compatibility with ComfyUI constraints.【F:src/comfyui_mcp/server.py†L90-L177】 - Expose configurable timeouts/retries, structured exceptions, and optional authentication header customisation for the HTTP client beyond the current static settings.【F:src/comfyui_mcp/client.py†L15-L59】 - Add structured logging (with correlation IDs/prompt IDs) and, optionally, metrics for request counts, durations, and failures.【F:src/comfyui_mcp/server.py†L27-L188】 ### §8 – Testing & Validation Strategy - Create unit tests for configuration loading, workflow parsing/semantic summaries, and mutation operations using representative fixtures (current suite only covers asset discovery).【F:tests/test_assets.py†L9-L56】 - Add integration tests with mocked ComfyUI endpoints covering submission, streaming, batch execution, and error paths.【F:src/comfyui_mcp/client.py†L26-L59】【F:src/comfyui_mcp/server.py†L53-L188】 - Provide end-to-end examples or notebooks demonstrating common interactions once APIs stabilise.【F:README.md†L1-L24】 ### §9 – Documentation & Onboarding - Expand `README.md` with setup instructions, configuration details, usage examples, and troubleshooting guidance beyond the current quick start bullets.【F:README.md†L1-L24】 - Add developer docs describing architecture, module responsibilities, and how to extend node heuristics or mutation primitives.【F:src/comfyui_mcp/workflow.py†L172-L295】【F:src/comfyui_mcp/mutations.py†L28-L148】 - Supply user-facing tutorials or walkthroughs for listing workflows, modifying parameters, and retrieving generated assets, potentially with screenshots or recorded sessions.【F:src/comfyui_mcp/server.py†L27-L188】 ### §10 – Future Enhancements & Roadmap - Plan for dynamic discovery of new custom nodes and automatic updates to the semantic registry once registry infrastructure exists.【F:src/comfyui_mcp/workflow.py†L172-L295】 - Explore richer diff visualisation (textual summaries or UI integration) to help users understand workflow mutations at a glance.【F:src/comfyui_mcp/mutations.py†L28-L148】 - Investigate session persistence or user profiles to remember preferred settings and assets across runs once the MCP surface is formalised.【F:src/comfyui_mcp/server.py†L53-L188】 - Prototype advanced batching features (e.g., grids, prompt matrices) after foundational batch support is implemented.【F:src/comfyui_mcp/server.py†L53-L81】 ## Recommended Near-Term Focus 1. **Broaden workflow understanding & mutations (Plan §§2–3)**: Deliver richer semantic indexing and stage-aware mutations so the server can accurately modify modern ComfyUI templates before adding advanced tooling.【F:src/comfyui_mcp/workflow.py†L172-L295】【F:src/comfyui_mcp/mutations.py†L28-L148】 2. **Improve execution robustness (Plan §4 & §7)**: Implement resilient streaming/output handling, retries, and structured errors to make execution reliable for agents.【F:src/comfyui_mcp/client.py†L26-L73】【F:src/comfyui_mcp/server.py†L53-L188】 3. **Ship MCP integration & schemas (Plan §5)**: Now that FastMCP tooling exists, define JSON schemas/enums for each tool so downstream LLM clients can validate inputs automatically.【F:src/comfyui_mcp/runtime.py†L1-L133】 4. **Establish automated validation (Plan §8)**: Add unit/integration tests to lock down behaviour before expanding features, and use fixtures to capture representative workflows.【F:tests/test_assets.py†L9-L56】 5. **Upgrade documentation (Plan §9)**: Provide actionable setup and usage guides to support early adopters and future contributors.【F:README.md†L1-L24】