generate_image
Generate images, videos, audio, 3D models, or image edits from a text prompt or existing image by building and enqueueing ComfyUI workflows, returning a prompt_id for async completion.
Instructions
Generate media from a prompt or an existing image — the high-level entry points that build the graph for you. Every action enqueues on the connected ComfyUI and returns the prompt_id immediately; the resulting asset_id arrives in the completion notification. Driven by the action parameter:
action:"image" — Text-to-image. Builds a txt2img workflow, filling any unspecified parameter from your configured defaults (get_defaults (action:"set") / COMFYUI_DEFAULT_* / config file), auto-selecting a local checkpoint when none is given — checkpoints known to lack a text encoder (e.g. video models) are skipped.
promptis required. For full control over the node graph, use create_workflow + enqueue_workflow instead.action:"audio" — Text-to-audio, supporting the ACE Step 1.5 and Stable Audio 3 model families. Builds the appropriate workflow graph, filling unspecified parameters from your defaults and auto-selecting local models.
model_family,promptanddurationare required. Requires a running ComfyUI with the corresponding model files installed.action:"video" — Text-to-video, or image-to-video when
imageis given (animate a start frame). Composes an LTX-2.3 distilled workflow on your LOCAL GPU using the render-verified Comfy-Org node stack (gemma text encoder + abliterated/distilled LoRAs). Needs the LTX-2.3 models (~24-46GB): install with apply_manifest --path packs/ltx-2.3-txt2vid/manifest.yaml (or ltx-2.3-img2vid for i2v); returns an actionable error if the checkpoint is missing.secondsis converted to an 8n+1 frame count. For i2v, higherstrengthmeans MORE adherence to the start frame but LESS motion (1.0 can freeze the clip) — keep ~0.6. This minimal path omits the synchronized audio + stage-2 spatial upscale that the full ltx-2.3 packs ship.promptis required. The video is written under output/video/ — find it with get_image (action:"list_outputs") (VHS/SaveVideo outputs may not appear in /history).action:"3d" — Generate a 3D model (glb/obj/fbx) from a text prompt or an input image, using the connected ComfyUI's hosted partner 3D nodes (Tripo, Meshy, Rodin, Hunyuan3D — auto-detected from the server; these are paid API nodes needing a comfy.org API key/login on the server or COMFY_API_KEY here).
modeis required ("text" needsprompt, "image" needsimage). Poll queue (action:"status") / get_history (action:"list") for the resulting model file (saved to ComfyUI's output directory). If the server has no 3D-capable API nodes, returns an actionable error naming local-pack alternatives.action:"controlnet" — Image conditioned by a ControlNet preprocessed image (pose skeleton, depth, canny, normal, etc.) plus a text prompt. Upload the control image first with upload_image (action:"image"), then pass its filename as
control_image.promptandcontrol_imageare required;checkpointandcontrolnet_modelauto-resolve from local models. control_image must ALREADY be a preprocessed map (this action does not run the preprocessor); requires a running ComfyUI with a matching controlnet model in models/controlnet/.action:"ip_adapter" — Image guided by a reference image's style/subject via IP-Adapter, plus a text prompt. Requires the ComfyUI_IPAdapter_plus custom nodes. Upload the reference first with upload_image (action:"image"), then pass its filename as
reference_image.promptandreference_imageare required;checkpointauto-resolves. Requires a running ComfyUI with ComfyUI_IPAdapter_plus and a matching IP-Adapter model installed, or the workflow will fail at execution time.action:"regenerate" — Re-enqueue the workflow that produced an EXISTING ASSET, optionally applying
overrides. Overrides are applied to any node input matching the key name (e.g. cfg, steps, sampler_name, scheduler, seed, denoise, text). Seeds are re-randomized by default so each call yields a fresh image unless seed is explicitly passed in overrides.asset_idis required. To re-run from execution HISTORY rather than a registered asset, use enqueue_workflow (action:"rerun").action:"upscale" — Upscale an image with an ESRGAN super-resolution model. Builds an UpscaleModelLoader → ImageUpscaleWithModel workflow (scale=2 supersamples the 4x result back down for sharper output) and enqueues it on your LOCAL GPU. Upload the source first with upload_image (action:"image") (or stage a prior output with upload_image (action:"stage")), then pass its filename as
image. Needs an upscale model in models/upscale_models/ (e.g. 4x-ClearRealityV1 / 4x_foolhardy_Remacri, provided by the anima/ernie packs or download_model); returns an actionable error if none is found.imageis required.action:"remove_background" — Remove an image's background, returning a transparent (RGBA) cutout. Builds a LoadImage → BiRefNetRMBG → SaveImage workflow using the ComfyUI-RMBG (BiRefNet) matting node and enqueues it on your LOCAL GPU. Upload the source first with upload_image (action:"image") (or stage a prior output with upload_image (action:"stage")), then pass its filename as
image. Requires the ComfyUI-RMBG custom node (pack: wan-transparent, or install_custom_node 'comfyui-rmbg'); the BiRefNet model auto-downloads on first run. If the node isn't installed, returns an actionable error telling you how to install it.imageis required.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| bpm | No | action:"audio" — TextEncodeAceStepAudio1.5 tempo in beats per minute (ACE only, 10-300, default: 120). | |
| cfg | No | CFG scale. Actions "image", "audio", "video", "controlnet", "ip_adapter" (video defaults to 1.0 for the distilled model). | |
| fps | No | action:"video" — frames per second (default 25). | |
| vae | No | action:"audio" — ACE VAE model filename (in models/vae/); auto-selected if omitted. | |
| clip | No | action:"audio" — Stable Audio CLIP encoder filename (in models/text_encoders/); auto-selected if omitted. | |
| mode | No | action:"3d" — "text" = text-to-3D from `prompt`; "image" = image-to-3D from an uploaded input `image`. REQUIRED for that action. | |
| node | No | action:"3d" — explicit 3D API node class_type to use (e.g. "MeshyTextToModelNode"); auto-selected if omitted. Use list_api_nodes with filter "3d" to see options. | |
| seed | No | Seed (omit to randomize). Actions "image", "audio", "video", "controlnet", "ip_adapter". | |
| unet | No | action:"audio" — ACE UNet model filename (in models/diffusion_models/); auto-selected if omitted. | |
| image | No | Filename of an image in ComfyUI's input dir (upload it first with upload_image (action:"image"), or stage a prior output with upload_image (action:"stage")). REQUIRED for action:"upscale" and action:"remove_background"; the start frame for action:"video" image-to-video; the input image for action:"3d" in mode "image". | |
| min_p | No | action:"audio" — TextEncodeAceStepAudio1.5 LLM min-p sampling (ACE only, 0-1, default: 0). | |
| model | No | Model file for the post-processing actions: action:"upscale" — an upscale model in models/upscale_models/ (auto-selected from local models if omitted); action:"remove_background" — the BiRefNet matting model (default 'BiRefNet_toonout'; auto-downloaded by ComfyUI-RMBG). | |
| scale | No | action:"upscale" — net upscale factor: 2 or 4 (default 4). | |
| shift | No | action:"audio" — ModelSamplingAuraFlow shift parameter (ACE only, default: 3). | |
| steps | No | Sampling steps. Actions "image", "audio", "video", "controlnet", "ip_adapter" (video defaults to 8 for the distilled model). | |
| top_k | No | action:"audio" — TextEncodeAceStepAudio1.5 LLM top-k sampling (ACE only, 0-100, default: 0 = disabled). | |
| top_p | No | action:"audio" — TextEncodeAceStepAudio1.5 LLM top-p nucleus sampling (ACE only, 0-2000, default: 0.9). | |
| width | No | Image width in pixels. Actions "image", "controlnet", "ip_adapter". | |
| action | Yes | What to generate. action:"image"/action:"video" require `prompt`; action:"audio" requires `model_family`+`prompt`+`duration`; action:"3d" requires `mode` (+ `prompt` or `image`); action:"controlnet" requires `prompt`+`control_image`; action:"ip_adapter" requires `prompt`+`reference_image`; action:"regenerate" requires `asset_id`; action:"upscale" and action:"remove_background" require `image`. | |
| clip_a | No | action:"audio" — primary text encoder filename (in models/text_encoders/); auto-selected if omitted. | |
| clip_b | No | action:"audio" — secondary text encoder filename (in models/text_encoders/); auto-selected if omitted. | |
| height | No | Image height in pixels. Actions "image", "controlnet", "ip_adapter". | |
| inputs | No | action:"3d" — provider-specific extra inputs passed through to the node (e.g. style, texture, quality). Use list_api_nodes (action:"schema") on the chosen node for valid keys. | |
| lyrics | No | action:"audio" — lyrics or song structure description (ACE only — section-by-section breakdown). | |
| preset | No | action:"ip_adapter" — IPAdapterUnifiedLoader preset (default 'PLUS (high strength)'). | |
| prompt | No | Positive text prompt. REQUIRED for actions "image", "audio", "video", "controlnet" and "ip_adapter"; for action:"3d" it is required in mode "text" and optional (passed through only if the chosen node accepts it) in mode "image". Unused by action:"regenerate", action:"upscale" and action:"remove_background". | |
| weight | No | action:"ip_adapter" — IP-Adapter influence on the output, typically 0.0-1.0 (default 0.8); higher = closer to the reference. | |
| sampler | No | Sampler name (e.g. euler, dpmpp_2m). Actions "image", "audio", "controlnet", "ip_adapter". | |
| seconds | No | action:"video" — clip length in seconds (default 4; ~10s max). | |
| asset_id | No | action:"regenerate" — asset id of the source generation. REQUIRED for that action. | |
| duration | No | action:"audio" — audio duration in seconds. REQUIRED for that action. | |
| language | No | action:"audio" — language code for prompt (ACE only, default: 'en'). | |
| strength | No | Two DIFFERENT knobs sharing one field, each with its own range, checked when the action runs: action:"video" (i2v only) — adherence to the start frame, 0-1 inclusive (default 0.6; higher = LESS motion); action:"controlnet" — conditioning strength, must be > 0, typically 0.0-2.0 (default 1.0; higher = stronger adherence to the control image). | |
| overrides | No | action:"regenerate" — map of input-name → new value applied to every node that already has that input. Common keys: cfg, steps, sampler_name, scheduler, seed, denoise, text. | |
| scheduler | No | Scheduler (e.g. normal, karras). Actions "image", "audio", "controlnet", "ip_adapter". | |
| batch_size | No | action:"image" — number of images to generate. | |
| checkpoint | No | Checkpoint filename; auto-selected from local models if omitted. The relevant checkpoint differs per action: a diffusion checkpoint for "image"/"controlnet"/"ip_adapter", the LTX checkpoint for "video", the Stable Audio 3 checkpoint for "audio". | |
| resolution | No | action:"video" — 'WIDTHxHEIGHT' e.g. '768x512' (rounded to multiples of 32; default 768x512). | |
| musical_key | No | action:"audio" — target musical key (ACE only, e.g. 'C major', 'E minor'; default: 'C major'). | |
| temperature | No | action:"audio" — TextEncodeAceStepAudio1.5 LLM sampling temperature (ACE only, 0-2, default: 0.85). | |
| weight_type | No | action:"ip_adapter" — IPAdapter weight mode (default 'standard' — required by current IPAdapter_plus builds). | |
| model_family | No | action:"audio" — audio model family; determines which workflow template and model loaders to use. REQUIRED for that action. | |
| audio_quality | No | action:"audio" — SaveAudioMP3 bitrate/quality (ACE and stable_audio_3, one of 'V0'/'128k'/'320k', default: '320k'). | |
| control_image | No | action:"controlnet" — filename of the (already-uploaded, already-preprocessed) control image in ComfyUI's input dir. REQUIRED for that action. | |
| timesignature | No | action:"audio" — TextEncodeAceStepAudio1.5 time signature (ACE only, one of '2'/'3'/'4'/'6', default: '4'). | |
| guidance_scale | No | action:"audio" — TextEncodeAceStepAudio1.5 cfg_scale, the text encoder guidance scale (ACE only, default: 2). | |
| filename_prefix | No | Output filename prefix. action:"audio" (default audio/ace_step or audio/stable_audio_3), action:"video" (default 'video/ltx-2.3') and action:"remove_background" (default 'ComfyUI_cutout'). | |
| negative_prompt | No | Negative prompt (default: empty / from defaults). Used by actions "image", "video", "controlnet", "ip_adapter" and — for the Stable Audio 3 family only — "audio". | |
| reference_image | No | action:"ip_adapter" — filename of the (already-uploaded) reference image in ComfyUI's input dir. REQUIRED for that action. | |
| controlnet_model | No | action:"controlnet" — ControlNet model file (in models/controlnet/); auto-selected if omitted. | |
| disable_random_seed | No | action:"regenerate" and action:"3d" — if true, do not randomize seed fields. For action:"regenerate", combine with `overrides.seed` to reproduce the exact original image. | |
| generate_audio_codes | No | action:"audio" — generate audio codes via the TextEncodeAceStepAudio1.5 LLM (ACE only, default: true). |