Prospero's Hoard

The Faustus button opens the shared copilot panel for the selected project. Faustus owns conversations, drafts, model selection and agent execution; Prospero supplies its native studio tools, project, current section and selected asset. Chats remain available in Faustus Studio, while completed turns refresh domain views through the existing data refresh flow. Creative engines and jobs retain their normal configuration. Configure the family Faustus endpoint in data/backend.json or PROSPERO_FAUSTUS_URL.

Such stuff as dreams are made on: can an agent direct a whole production?

A local media studio that drives your ComfyUI, ffmpeg and a local TTS to make consistent characters, photocards, album art and music videos cut on the beat, by hand or entirely over MCP, and remembers exactly how every asset was made.

Español · Quick start · Real production example · Connect to Faustus · MCP reference · Voice studio · Portfolio

Actual application, synthetic demo data. Every picture comes from the bundled demo backend, a procedural stand-in for ComfyUI that draws labelled placeholder scenes; with your ComfyUI connected the same screens show real model output.

Why

A language model asked to help with a small music production (an invented group, its photocards, a cover, a video for the single) can only describe what it would do. It cannot run a node graph, it cannot keep a character's face consistent across forty images, it cannot find the beat of a song, and nobody can later answer "which seed and checkpoint made this card?". Doing it by hand means juggling a node editor, an image editor, an audio tool and a video editor that share no memory.

Prospero turns each step into a typed, queueable operation with a recorded recipe: characters are mentioned as @Name and their description is inlined every time, every asset keeps the exact parameters that made it (and can be re-run to the byte on the same backend), songs are analysed for tempo and sections, and a timeline is cut on the beat and rendered with ffmpeg. The local model directs; the studio does the work and reports back with ids and pictures.

Actual application, synthetic demo data: the auto-cut of the 30 s demo song (120 BPM, quiet-loud-quiet), two beats per shot in the loud section, four in the quiet ones, and the preview it rendered with ffmpeg.

Related MCP server: Compeller MCP

Use cases

• An independent musician with a finished single imports the song, lets the beat tracker find tempo and sections, times the lyrics, and gets a cover, a lyric card and a 9:16 video cut on the beat - without opening a node editor or a video editor.

• A writer or game designer with an original cast gives each character a look prompt and a canonical reference, then asks for "@Iris and @Mika backstage": the same faces come back across dozens of images (consistent=true), and every card says which seed and checkpoint made it.

• A local model in Faustus asked to "make a teaser for the single" calls studio_generate_image, studio_timeline and studio_render, gets short ids back, looks at a picture only when it asks for one, and every call it made is listed under Assistant activity.

• Someone who explains things for a living types a topic ("why the sea glows at night") and gets a narrated vertical short: a script with a hook written by the local model, a voice-over, captions that light up word by word, stock footage or generated pictures cut to what is being said, a music bed that ducks under the voice, and the title, hashtags and credits ready to paste.

• A ComfyUI user with their own workflows imports the UI-format export, gets it converted, checked against the live node list and mapped to named parameters, and from then on generates with it from the studio or from an agent, with lineage.

What is implemented

The studio combines image and video creation with optional identity, outfit, setting, style and pose references, a large monitor, one-click media viewing and take comparison. Text and shots imports TXT/LRC/SRT/VTT and links phrases and timing to the storyboard; an idea remains enough to start. The searchable tool drawer retains the complete toolset. Timeline rendering no longer silently freezes the last frame to cover substantially short clips. See redesign research and local tests.

Actual application, synthetic demo data: the photocard set rendered for the five invented members, opened in the lightbox with its recipe and inputs.

Actual application, synthetic demo data: the "Reference film" space after one run - three sheets feed a frame through its references, the frame starts a draft clip, the style text reaches both.

Models

Wan Animate 2 ships as bf16 (33 GB) and int8: cast the bf16 to fp8 with python -m prosperos_hoard.devtools.cast_fp8 wan_animate_2_distill_bf16.safetensors ComfyUI/models/diffusion_models/wan_animate_2_distill_fp8_e4m3fn.safetensors (run it with ComfyUI's Python, which has torch) (transformer-block weights only, a minute); Prospero loads the fp8 file when it is there. A template can name the card it needs (min_card_mb): with a render pool, a job waits for a server whose card holds it and the others keep taking the rest. Animate renders 3 s (73 frames at 24 fps) by default: on a 16 GB card 81 frames already spill out of VRAM and crawl. Prospero's launcher starts ComfyUI with disk-backed offload off (a big model then streams from RAM, not from the disk, on every step); on a Windows card that also draws desktop apps, "comfyui": {"args": ["--reserve-vram", "2.5"]} in ~/.hoard/backends.json keeps them from pushing the render into shared memory.

studio_generate_image's engine parameter (and a project's own image_engine setting) picks between the image families: auto (default) resolves to Qwen-Image 2.1 when its node class and model files are installed, else Flux schnell, else SDXL - every result says which one it actually used. A template name always wins over engine when both are given. See docs/ARCHITECTURE.md for how the converter turns ComfyUI's own UI-format export of each template into the API-format workflow above, and how a missing model file is reported.

Upscale and background removal

Two edits of studio_edit_image (and the lightbox buttons Upscale x2, Upscale x4 and Remove background) run on ComfyUI core nodes only, with no prompt, and work on any image asset:

• upscale enlarges with an ESRGAN-family model (template esrgan_upscale). The model works at 4x and the result is scaled down to the requested scale, 2 or 4 (default 2). Any other value is refused (bad_parameter), and so is a result larger than 8192 px on a side (too_large; the message gives the source size and the maximum). model may name another file that is installed in upscale_models; the default is RealESRGAN_x4plus.safetensors. A source with transparency (a cut-out) runs esrgan_upscale_alpha, which puts the source alpha back at the new size, so it stays a cut-out.

• remove_background cuts out the subject with BiRefNet (template birefnet_remove_background, the same graph as ComfyUI's own "Remove Background (BiRefNet)" blueprint) and imports a PNG that keeps its alpha channel. The model file is birefnet.safetensors unless backend.json names another one under "bg_removal_model".

Both always produce one image and record the source in the recipe (edit_image:upscale, edit_image:remove_background). The model files are not bundled; put them here (folders relative to your ComfyUI install):

When a file is missing the job fails with model_missing, naming the folder and the files ComfyUI does list. Both nodes need a recent ComfyUI (the background-removal loader is not in older releases); an older one reports the missing nodes. VRAM estimates are 2500 MB (esrgan) and 3500 MB (birefnet), editable like the others.

Quick start

git clone https://github.com/Luissalet/ProsperosHoard.git
cd ProsperosHoard

You need Python 3.11+ (3.13 recommended), Node.js 22 for the interface and ffmpeg on PATH (or the bundled imageio-ffmpeg binary is used). ComfyUI is optional: --demo runs everything against a procedural stand-in.

Windows

Double-click Iniciar Prospero's Hoard.cmd (or run scripts\start.ps1). The first run creates .venv with Python 3.13, installs requirements-lock.txt, builds the interface with npm and opens http://127.0.0.1:8815; later runs reinstall only when the lock file changed. Detener Prospero's Hoard.cmd stops it (only after checking that the port really belongs to Prospero).

Manual steps:

C:\Python313\python.exe -m venv .venv
.venv\Scripts\python.exe -m pip install -r requirements-lock.txt
cd frontend; npm ci; npm run build; cd ..
.venv\Scripts\python.exe -m prosperos_hoard            # real data in .\data
.venv\Scripts\python.exe -m prosperos_hoard --demo     # demo data in .\data-demo

Linux / macOS

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-lock.txt
(cd frontend && npm ci && npm run build)
.venv/bin/python -m prosperos_hoard --demo --no-browser

The first --demo run seeds its data (about a minute on two CPUs) before the server answers. Then open http://127.0.0.1:8815 (curl http://127.0.0.1:8815/api/health answers "service": "prosperos-hoard").

Flags: --port, --data-dir (or PROSPERO_DATA_DIR), --demo, --no-browser. PROSPERO_ALLOWED_HOSTS (comma-separated) lets the API answer to a LAN or tailnet name. --demo starts the procedural demo backend (with PROSPERO_DEMO_MOTION=1 its fake ComfyUI also lists the big video models - Wan 14B, VACE, Bernini-R - so those features can be tried), seeds an original five-member group with portraits, stage shots, a cover, a photocard set, a synthetic 30 s song with timed lyrics and an auto-cut timeline, and renders its preview video. To use your ComfyUI, leave it on 127.0.0.1:8188 (Hoard Link finds it) or set its URL in Settings. On a GPU shared with a language model, renders slow down a lot; PROSPERO_COMFY_TIMEOUT_S raises how long a job may run (defaults: video 1 h, audio 30 min, image 20 min).

Actual application, synthetic demo data: the synthetic demo song analysed by the built-in beat tracker, with the section estimates and the LRC lyrics timed to it.

Connect it to Faustus

Prospero's Hoard is a plugin for Faustus, the local AI workspace, and declares itself with faustus-plugin.json. Start it, then in Faustus open Connectors -> Nearby apps -> Add: Faustus finds it on port 8815, reads the manifest and launches the MCP adapter (prosperos_hoard/mcp_server.py, stdio). The model backend is shared: the studio asks Hoard Link which ComfyUI and TTS are already running instead of loading anything of its own.

MCP tools

| production_export_lumiere | Write a production's cut as FCP7 XML + EDL and open it as a project in Lumiere's Hoard | no | | cast_import_character | A cast member from a name, a description, a look and reference images (the same name and source_ref twice is the same member) | no | | production_from_storyboard | A production draft from shots {text, duration_s?, image?}; not queued, it still needs a song | no | | voice_tts | Speak a text with a saved voice, a chosen engine (engine) or the best installed one, at a speed of 0.5 to 2.0; returns the path of a WAV | no |

It works with any MCP client over stdio too:

{"mcpServers": {"prosperos-hoard": {
  "command": "C:/.../Prospero's Hoard/.venv/Scripts/python.exe",
  "args": ["C:/.../Prospero's Hoard/prosperos_hoard/mcp_server.py"],
  "env": {"PROSPERO_URL": "http://127.0.0.1:8815"}}}}

Arguments, result shapes and limits of every tool: docs/MCP.md. The end-to-end recipe the agent follows: skills/idol-production/SKILL.md.

The Hoard family

Prospero joins the other Hoard apps through the shared family contract (the vendored hoard_link):

• Shared agent contract. GET /api/agent/tools lists every per-tool route (query and body arguments, readOnlyHint, the same descriptions as the MCP adapter) and POST /api/agent/call {name, arguments} runs one with the family bearer token (data/mcp-token); it runs the very function the per-tool route runs. Both are placed before the catch-all routes. /api/health has a hoard_link block. faustus-plugin.json declares it.

• Accountable agents (HoardLink 0.8.2). Every call that changes something through POST /api/agent/call must say why (reason, 3-300 characters, in the body or among the arguments; otherwise 400 reason_required and nothing runs), is written to data/agent_journal.jsonl with the agent (X-Agent-Id) and session (X-Agent-Session) that made it (secrets masked), and a whole session can be taken back: POST /api/agent/undo {session, dry_run: true} shows what would happen, {session, confirm: true, reason} does it. Only that session's writes are touched, and a write another session or you changed afterwards is left alone and reported as a conflict. The web interface and the per-tool routes are the person's way in and need no reason. Details, the list of tools that can be undone and the token profiles: docs/MCP.md.

• Job events. Renders, songs, clips and whole productions send prospero.job.queued|started|progress|done|failed|cancelled with job_id, title, kind (render, song, clip, production), progress, gpu, url; progress is throttled to one event every 5 s. A production that stops for you finishes its job as done with status: "awaiting_review" and awaiting (take, animatic, script or review).

• Notifications. Only productions, the thing that waits for a person: it needs a take, the animatic or a review, it finished, or it failed (failed = high priority). They go through the family hub and link to the production's page. Settings > Production notices (or GET|PUT /api/family/settings): notify.via = auto (the hub when it answers), hub (always try) or off; notify.language = es or en. Stored in data/family.json.

• GPU lease. A job in the GPU lane holds the hub's lease for its class of model (hoard_link.lease) on top of the existing VRAM check (sized by edit operation - pose and depth maps 3 GB, upscale, background removal - and by clip template - InfiniteTalk, first-last-frame 14B, S2V - rather than by the image engine), so the editor's transcription and the language models do not load at the same moment. With no hub it changes nothing; when the hub keeps the lease queued past two minutes the job goes to waiting_gpu and retries. PROSPERO_GPU_LEASE=0 turns it off.

• For Lumiere. On a production's page, Export to Lumiere (also the tool production_export_lumiere) writes the cut's XML and EDL to data/exports/lumiere/ and asks Lumiere's Hoard to open it as a project; if it is not running or may not read that folder, the answer says so and the files stay there. Lumiere may only read some folders (LUMIERE_FILE_ROOTS): allow Prospero's data folder.

Shared models (HoardLink)

Prospero hosts no model. It asks HoardLink (vendored in prosperos_hoard/hoard_link/, a byte-identical copy of HoardLink 0.8.0) for the image/video backend (ComfyUI) and tts (Faustus TTS, with Piper as the local fallback), the same resolver every Faustus plugin uses. Resolution order in one line: an explicit override in Settings, data/backend.json or a HOARD_* environment variable, then a running Faustus instance's own registry, then a server already listening on loopback (ComfyUI on 8188). Music comes from ACE-Step through the same ComfyUI, or from a small documented HTTP server you point HOARD_MUSIC_URL at. The Backends screen and studio_status always say what was found and why; nothing is loaded or unloaded behind your back.

What comes from the family commons

The shared library (HoardLink 0.8) replaces code this app used to carry:

• Downloads from a link (Library > Download from a link, studio_download_media) go to the family's link downloader (Links Hoard) first, with the same limits as before (sections, 20 minutes, 1080p). Only when it cannot be reached does Prospero run yt-dlp itself, found with the shared tool finder (its own binary, or python -m yt_dlp). Links are checked with the shared public-address rules: private, loopback, link-local and metadata addresses, numeric hosts and URLs with credentials are refused. Stock footage downloads (Pexels, Pixabay) get the same check on every redirect, a connection pinned to the checked address and a 400 MB cap.

• Speech to text (the faster-whisper engine of the voice studio) asks Funes's Hoard first (one Whisper for the family) and otherwise uses the shared transcriber here, on the GPU when the hub lends it, with Whisper's inventions over silence filtered out.

• Speech for other apps: voice_tts takes engine (an installed engine id; a saved voice's own engine wins) and speed (0.5 to 2.0).

• ffmpeg, ffprobe are found by the shared finder (HOARD_FFMPEG, PATH, the usual Windows folders, then the bundled wheel); child processes are the shared ones (a timeout or a cancelled render or training kills the whole process tree).

• Subtitles and lyrics: SRT, VTT, TXT, LRC and the ASS text escape are shared (empty cues are skipped; LRC import reads several stamps per line, [offset:] and enhanced word tags). Settings, a production's state and recipes are written atomically with Windows lock retries; ids are the shared ULIDs.

• The guard in front of the API (Host, Origin, cross-site checks, websockets too) is shared: a rejected request is a 403 with a short message. PROSPERO_ALLOWED_HOSTS adds LAN or tailnet names (studio.lan, *.ts.net). A cross-site request that is not a page navigation is refused. The MCP bridge ignores proxy variables and the longest server-side wait is 150 s (wait_s).

• Production notices use the shared auto | hub | off router (a notice the hub holds for quiet hours counts as delivered).

A music video from the app

Productions > New music video: a title, the lead from the cast (its canonical image keeps the look) or a new one, the concept, the song (composed - sound, lyrics language, length - or one you already have, with its lyrics), how many shots, which shots get Wan clips (every moving shot, only the lead's, or none for a cut of stills) and the formats. Plan with the local model writes the shot list, the shared look and, for a composed song, the tags and the lyrics - in the language chosen (an answer in the wrong language is asked again once, then flagged) - with thinking off and a long timeout, so a 27B split over four cards answers in a minute or two. The button counts the time and Stop waiting gives up on it. The answer is streamed: a model that does not start writing, or goes quiet, for four minutes (typically because a render has filled the GPUs it shares) ends the plan with a message naming the render in the way, instead of hanging. A reply the planner cannot use (no shots, another shape, cut short) is asked for again once, with room for every shot; the last unreadable one is kept in data/logs/plan_last_bad_reply.txt. A render that runs out of time is stopped on ComfyUI too, so it does not keep the card busy for an image nobody collects. The plan is then reviewed (critic, on by default in studio_video_plan): a rubric that needs no model (near-duplicate shots, one shot size everywhere, the lead never or always on screen, the same motion everywhere, song sections without a shot) and a director's pass that rewrites the weak shots. In the dialog the first list shows at once and the review runs on its own (POST /api/productions/plan/critique) while you already edit it: its notes come in the app's language (notes_language) and its rewrite is a proposal, Apply or Keep mine; once applied, Use the first list brings back the list before the review (draft.first_shots). Everything is editable before anything renders; I'll write the shots skips the model. Create the production runs the usual pipeline: the lead, the song takes (with more than one, it pauses so you listen and Use this take), Qwen-Image stills from the canonical image, the animatic to review, the Wan clips and the cut on the beat. In the viewer, Animate makes a Wan 2.2 clip (SVD only when Wan is not installed) and Edit with the instruction edits the picture with Qwen-Image keeping its subject.

The production page keeps the shot list editable the whole time: every shot shows its still, its song section, where it plays in the song (from the animatic's cut) and, while rendering, how it is going ("still · 1:23 / ~2:00", queued, failed). Click a shot to change what it shows, the motion text, whether the lead is in it, its section (with that section's lyrics next to it), still or clip, which variant is the still, or to regenerate or delete it; + between two shots adds a new one there. Each shot can take its own references, each with what to take from it ("copy this dance pose", "these Pokemon dance in the background", "this place"): from the library, frames taken out of a video or an animated GIF, an upload, or a YouTube / X / Instagram link (only the part between from and to is downloaded). They follow the lead's canonical image as <image2>, <image3>... Edits apply when the production is paused (Pause to edit keeps whatever already rendered); Save and render rebuilds only what changed. Lyrics gives an existing song its lyrics with [Verse]/[Chorus] tags: timed to the song, they make each shot play over its section. A lead shot is edited from the canonical image keeping its design (face, body shape, colours, props) but taking the pose and action of the shot's text. References shape the still; the clip animates that still from the motion text with Wan 2.2 I2V 14B when it is installed (camera presets in the editor: orbit, crane from feet to head, push-in, tracking, low angle, handheld), else Wan 2.2 TI2V 5B.

Motion from a video: a shot can take a video (library, upload or a link, from a chosen second) as its motion reference. Its clip then runs Wan Animate 2: the character of the shot's still performs that motion - a dance, a gesture, a stunt - frame by frame, no skeleton needed, with the background from the shot's text. The video is only a source of movement: nothing of its picture ends up in the clip.

Background cast: one image and a name per character allowed in the background (up to 24). Shots ticked Background from the cast take a few of them (three by default, rotating so everybody shows up; or exactly the ones picked in the shot) as references, with the instruction that only they appear behind the lead, exactly as drawn, nobody invented. Changing the cast redraws only the shots whose crowd changed. When the cut uses clips it cuts on whole bars about every 4 s (2 s in the chorus), so a dance has time to read (clip_settings.cut_s, or explicit beats_* options).

Library > Download from a link brings a video (or just a section of it, or its audio) from YouTube, X, Instagram and the other sites yt-dlp knows, as an mp4 in the project (whole videos up to 20 minutes, or the section between two times), through the family's downloader when it runs and with yt-dlp here otherwise. Animated GIFs import as videos.

Backends > These ComfyUI servers are only for Prospero: when a server is idle, a job may load its model in place of the last job's instead of waiting for free VRAM (off by default, for GPUs shared with other apps). A model estimated bigger than the card never waits for more than 85% of it: ComfyUI offloads the rest.

Running without Faustus

The header carries the machine's vitals on every screen: GPU use, one tank per card (as wide as the card is big), VRAM, the hottest card's temperature, RAM and CPU. Click it for every card (use, temperature, power, VRAM and the servers and models on it), RAM with the committed memory (RAM + page file, what a model that does not fit ends up using) and CPU, with Free ComfyUI and Stop for the servers that may be stopped.

Bad results go: Select in Generate and in the Library, then Delete (the Delete key in the Library too); Delete in the viewer (or the Delete key twice) moves to the next one. Deleted assets go to the trash (files under data/trash/<id>/) with ten seconds of Undo, and Library > Trash restores them or empties it for good. An asset used as a cover, a character's canonical or reference image, a group logo or on a board is refused with the reason and a Detach and delete (restoring puts it back); one used in a timeline is always refused.

The Generate screen opens with the model header: the image engine (Automatic = Qwen-Image 2.1 when installed, FLUX.1, SDXL, SD 1.5, your imported workflows) and the model file for it, where ComfyUI runs (with Start when it is off and Free ComfyUI), and whether the engine fits in the GPU it will use: "Qwen-Image 2.1 needs about 12 GB; GPU 3 has 3 GB for it... held by llama.cpp (qwen3.8-27b)" with a Stop llama.cpp button right there. GPU memory lists every card with what each server keeps loaded (ComfyUI, llama.cpp and its model, Ollama and its models) and the start/stop actions. With Qwen-Image or Kontext the reference images go to the engine's own edit (up to 10 with Qwen, <image1> is the subject) at full denoise; the img2img strength and the sampler settings only apply to SDXL, and empty fields under Advanced use each engine's tuned values. An edit keeps the size and framing of <image1> (Like , picked as soon as the first reference goes in; a ratio forces that exact canvas instead), and typing < in the prompt lists the references (<image1> with its thumbnail...; Tab or Enter inserts one), and with several references the panel warns when the instruction does not name one of them - say what to take from each, e.g. <image1> wearing the jacket from <image2>, and describe it by what the reference shows rather than restating its colours, which the edit follows over the image. A ComfyUI busy with a heavy render shows as running (busy), not off.

Prospero does not need Faustus running. Backends > Local services lists the servers it uses - the main ComfyUI, every render-pool server, Ollama and any server you describe in ~/.hoard/backends.json - with a Start button (and a GPU choice for ComfyUI: the card with the most free memory by default) and a Stop button for the ones the Hoard family started. With "Start ComfyUI by itself when a job needs it" on (the default), a render queued while ComfyUI is off starts it first and then runs; with it off, the job fails with a message pointing at the Start button. Starting ComfyUI loads no model: memory is only used when a job runs. The agent does the same with studio_services, studio_service_start and studio_service_stop.

ComfyUI is found in COMFYUI_DIR, the folder saved in Backends, or the usual places (D:\LocalAI\ComfyUI, C:\ComfyUI, ~/ComfyUI, the portable build...), with the Python of its venv, .venv or python_embeded. It is started on loopback with --cuda-device for the chosen GPU; a server on any port other than 8188 gets its own output, temp, user and database folders under ~/.hoard/backends/, so two instances never race. Ollama is found on PATH or in its default install folder. The launcher is hoard_link/launch.py from HoardLink, shared with Hoard Hub and the rest of the family: ~/.hoard/backends.json says where things are installed, ~/.hoard/backends/state.json which processes the family started (pid and creation time, logs next to it). A ComfyUI started from Hoard Hub shows up here and can be stopped here, and the other way round; a server started by hand or by Faustus is shown as running and never stopped.

{
  "comfyui": {"dir": "D:/LocalAI/ComfyUI", "gpu": "auto", "args": ["--lowvram"]},
  "ollama": {"exe": null},
  "commands": [
    {"id": "llamacpp", "label": "llama.cpp", "argv": ["powershell", "-NoProfile", "-File", "D:/LocalAI/Start-LlamaServer.ps1"],
     "health": "http://127.0.0.1:8081/health", "capabilities": ["llm", "vision"]}
  ]
}

Production example

scripts/productions/no_mires_atras.py produces a single by FAROL, "NO MIRES ATRÁS" in Spanish or, with --lang en, "DON'T LOOK BACK" in English (an original night creature: a paper-lantern head, always still, always a little closer), end to end through the MCP adapter only - it spawns mcp_server.py over stdio, the same path Faustus uses, and fails if any result carries a picture it did not ask for. Nine steps: the project (with an --engine auto|qwen21|flux choice, set as its own default); a turnaround sheet cropped to its front view as FAROL's canonical reference; the song with ACE-Step; twelve stills (an edit template with that reference when FAROL is in frame, a fresh txt2img in the same night look when not - Qwen-Image 2.1 when installed, else Flux); Wan clips from the best stills; a solo photocard set in five idol looks with its contact sheet; the "night" cover, tracklist back, teaser poster and lyric card; lyrics timed to the song's bars; 9:16 and 16:9 cuts that follow a storyboard per section and change shot on every sung line, graded sodium-night with grain, vignette and a flash + colour-split glitch on the chorus downbeats, horror karaoke captions; and a REPORT.md with every asset id, timings and what to review. It checkpoints every still, clip, card and render, so an interrupted run resumes where it stopped; --only <step> redoes one step.

On Windows, with the app running and ComfyUI started on a 16 GB card:

# the demo backend (no GPU), a few minutes
.venv\Scripts\python.exe scripts\productions\no_mires_atras.py --backend fake --quality draft
# the real thing against the running app and ComfyUI
.venv\Scripts\python.exe scripts\productions\no_mires_atras.py --backend real --quality final
# pin the image engine instead of "auto" reaching for Qwen-Image 2.1 first
.venv\Scripts\python.exe scripts\productions\no_mires_atras.py --backend real --quality final --engine flux
# the English version on top of a finished run: same pictures, a sung song, a clip for every shot
.venv\Scripts\python.exe scripts\productions\no_mires_atras.py --backend real --quality final --lang en --reuse-from data\productions\no_mires_atras --motion full --song-takes 4
# after re-timing the lyrics by ear in Audio > Lyrics timing and exporting the LRC
.venv\Scripts\python.exe scripts\productions\no_mires_atras.py --backend real --quality final --only timeline --lrc-path C:\Users\<you>\Music\no_mires_atras.lrc

In the app: productions and recipes

The same pipeline also runs inside the app as a production (studio_production_create, or the Productions screen): one orchestrator job that queues the reference sheet, the song, every still and every clip as ordinary jobs (a render pool renders them side by side), checkpoints each item into data/productions/<slug>/state.json, and writes a REPORT.md. After the stills and the song it makes an animatic (the stills cut exactly like the final, 720p, with a plan.json and the GPU minutes the clips will cost) and waits for your review before rendering a single clip. It resumes where it stopped after a failure, a cancel or a restart; studio_production_shots swaps a still for another variant, turns a clip on or off or rewrites a shot, and only what depends on it is redone.

A finished production - including one made by the script above - becomes a recipe: studio_recipe_export(production, name) writes data/recipes/<name>.json with every stage, prompt, seed, template and setting, and the lead abstracted into a {lead} casting slot ({lead}, {lead.look}, {lead.negative}, {lead.palette[0]}...), plus warnings for shot prompts that still describe the old lead's props. studio_recipe_run(recipe, cast={"lead": <character id or {name, look}>}) ("recreate this with X", or Recreate with... on the Productions screen) starts a new production with the slot filled: an existing character keeps its canonical reference, a new one gets a reference sheet first, and the song (unless its lyrics name the old lead) and the stills and clips of the shots the lead is not in are reused (options.reuse: ["song", "frames", "clips"]).

A production's page works like the studios it borrows from: a pipeline of its stages (done, running, paused for review, failed) and one banner with the next thing to do - pick the song take (the takes play right there), watch the animatic and render the clips (with the GPU minutes it will take), resume after a failure (the error in plain words, the raw text one click away) or continue after an edit - and four tabs: Storyboard (shots and the song track), Preview (final cut and animatic), QA and History. Each project has its own Videos page (a gallery with a cover, the stage and a progress bar per video) and its Overview shows them with a "New music video" button that makes it in that project. The header's "N active" button opens what is running now - with progress, a cancel, and a click through to the production it belongs to. Dialogs keep their buttons in view however long they get.

Autopilot (a switch on the production, or when creating it) runs every stage without stopping - the first song take, no animatic review - and the creation dialog shows what the run will cost on this machine (stills, clips, GPU hours) before anything starts. Before a run, the page checks what would stop it - ComfyUI down (with a Start button), no music model for a song still to compose, no ffmpeg, no GPU with enough free memory - and says so in plain words. A finished cut downloads for Premiere Pro or DaVinci Resolve (FCP7 XML + CMX 3600 EDL, with timed captions as SRT/VTT and word/karaoke metadata as JSON when present; the media are referenced where they live on this computer), from the production's Preview tab or the Timeline screen (studio_export_timeline).

Import the SRT or VTT separately as a subtitle track: XML markers and EDL comments are labels. Caption text and timeline timing transfer; font, placement and animation do not. The export clips caption ends to the cut without changing the stored timeline.

The Audio screen composes songs too (style, lyrics, BPM, length, key, takes) and imports audio files; a character without a reference image gets a "Make a reference" button that opens Generate with a portrait prompt, and any picture in the viewer can become a character's reference image in one click.

Song and lyrics on a production is a video editor standing up: the lyrics timed to the song on one vertical line (sections beside them, a playhead while it plays) and a shot track next to them. Shots are laid one after another - dragged in from the shot list, or Add next - then moved or trimmed at either edge (they snap to the lyric lines; Alt places freely, and they snap to the song's beats too: bar lines are drawn across the track), and each block shows the words it plays over and how many bars it lasts. Where the cut put the shots left off the track shows as faint marks; click one to pin that shot there. A placed shot plays exactly there in the animatic and the final cut (span {start_s, end_s} on the shot; spans cannot overlap); shots left off the track are placed by the cut by their section. Clicking lines (shift for several) makes a new shot for exactly those words. Change song swaps the song without redoing the stills or clips: a song from the library of any project (with its own lyrics when it was composed here), an uploaded file, another take, or a recompose with new style/tempo/length/words - the lyrics are timed to it at once. studio_production_song and studio_production_timing do the same for an agent. The library pickers (references, motion videos, background cast, songs) are a searchable menu with a preview: click to see it (videos and songs play), double click or Enter to use it, "All projects" to look beyond this one.

Deleting a project (the bin on its card in Projects, or studio_delete_project) moves it to the trash with everything in it and its productions: it disappears from every list and from the all-projects search. "Deleted projects" at the bottom of Projects restores it as it was, or deletes it for good (rows, files, thumbnails, trashed assets and its productions; no undo). A project with a job queued or running is refused.

A production whose run died without writing its end (the app closed, or Windows refused the state write because another reader had the file open) no longer stays "running": it shows as failed, can be edited and resumed. State and recipe writes retry the replace for a moment on a Windows sharing violation instead of failing the run.

Narrated shorts

studio_short_create(topic="why the sea glows at night", options={"language": "en", "visuals": {"source": "auto"}, "music": {"mode": "compose"}}) (or New short on the Productions screen) makes a vertical video out of a topic, as a production of its own kind (kind: "short") with the same resume, jobs and lineage:

1. script - the local model writes a hook and 5-9 segments, each with the narration, an English image prompt and English stock keywords, plus a title, a description and hashtags; or pass your own script (plain text, one paragraph per segment, or segments). With settings.script_review it stops here for you to read it.

2. narration - each sentence voiced and placed on one clock (exact sentence and segment times); with faster-whisper installed, the word times it hears are aligned onto the script's own words, so the captions keep your spelling.

3. music - none, an existing asset, an ACE-Step instrumental, or a track picked from data/music/.

4. pictures - each segment split into ~3 s shots, filled with stock footage (Pexels/Pixabay, needs a free key in Settings) or generated stills in the project's image engine; visuals.clips animates the longest stills with Wan after an animatic review.

5. mix - the voice over the music, which a sidechain compressor pushes down while someone speaks, normalised to -14 LUFS.

6. cut and render - the shots on the narration's clock, captions two or three words at a time with the spoken word highlighted, every aspect you asked for; then REPORT.md and publish.txt (title, description, hashtags and the footage credits).

studio_production_script(production, script) replaces the script and redoes the narration, pictures, mix and render (the music is kept); count=3 makes three variants with other seeds (other footage, other generated pictures and, from a topic, another script).

The real run

The same script ran against a real ComfyUI 0.37 on 16 GB cards, in two versions that share every picture and clip. DON'T LOOK BACK (--lang en --motion full) is a sung English horror anthem told by the creature, with a clip for every shot, rendered on a render pool of three ComfyUI servers, one per card. NO MIRES ATRÁS is the first take, a Spanish horror rap cut from seven clips on one card. Qwen-Image 2.1 made the reference sheet, the stills and the photocards, Wan 2.2 TI2V 5B the clips and ACE-Step 1.5 the songs. The canonical reference, the best variant of each shot and the song take were picked by eye and by ear. The lyrics were aligned to the vocals with faster-whisper (outside Prospero) and imported as an LRC with timed section markers. Everything below comes straight out of the app, only downscaled for this page: nothing was retouched. The prompts, seeds, settings, timings and the nine problems the run found (all fixed) are in the full example.

On 16 GB cards: 48 stills in 57 min, the first seven 5 s clips in about 70 min on one card and eleven more in about 40 min on three, four song takes in 2 min, and both cuts (preview and 1080p final) in about 7 min.

Architecture

FastAPI + SQLite (WAL, one connection per thread) with a GPU worker, a CPU worker and an orchestrator thread (whole productions) over a persistent job table; business logic in plain modules (engine, comfy_driver, design, audio, timeline, video, voices, productions, recipes) with no web imports; Hoard Link vendored for backend resolution; the MCP adapter is a separate stdio script that only speaks HTTP to the app.

flowchart LR
  UI["React UI"] -->|"/api/*"| API["FastAPI app<br/>127.0.0.1:8815"]
  MCP["MCP stdio adapter"] -->|"/api/agent/*"| API
  API --> DB[("SQLite<br/>projects, assets, lineage, jobs")]
  API --> JOBS["GPU + CPU job workers"]
  JOBS --> COMFY["ComfyUI<br/>(image, video, music)"]
  JOBS --> FF["ffmpeg<br/>(renders, finishing)"]
  API --> DESIGN["Pillow designer"]
  API --> AUDIO["beat tracker"]
  API --> LINK["HoardLink"] -. "resolves" .-> COMFY
  LINK -. "optional" .-> TTS["Faustus TTS / Piper"]

Details, data model and decisions: docs/ARCHITECTURE.md. Every endpoint: docs/API.md.

Privacy and security

Local only: the app binds 127.0.0.1, sends no telemetry and only goes online when you ask for a Piper voice that is not downloaded yet (from the rhasspy/piper-voices repository on Hugging Face). Requests with a foreign Host header are refused (DNS rebinding) and so are writes from other websites (Origin and Sec-Fetch-Site checks). Agents can import files only from your home folder, data/inbox and folders you add in Settings, and only files whose content matches their type; files are always served by id, never by a path the client sends. Every call an assistant makes is written to the agent_calls audit table and shown under Assistant activity (tool, argument summary, duration, result). The demo uses invented people only. Voice cloning runs entirely on local engines that you install explicitly (nothing is downloaded silently); a cloned voice is only ever created from a sample you provide, and using someone else's voice without their consent is on you, not the tool. Data stays in data/ (or your --data-dir); the Faustus token is stored in data/backend.json and never returned by the API.

Development

.venv\Scripts\python.exe -m pytest -q
cd frontend; npm ci; npm run build

On Linux/macOS the same with .venv/bin/python. about 675 tests pass, offline, with no GPU and no model downloads (the demo backend stands in for ComfyUI, and the voice studio's own suite adds fake TTS/STT engines plus real, optional tests against Piper and faster-whisper when they are installed). The suite covers the voice engine registry (install-hint status without ever importing a heavy dependency), the voice library's sample processing and quality check, the audiobook chunking/assembly and dubbing pipelines end to end (including the time-fitting maths, glossary-aware translation through a fake local model, and per-segment re-synthesis), the /api/voice/* HTTP surface, and the 9 voice MCP tools driven over the real MCP protocol; and the MCP protocol end to end (the adapter spawned over stdio against a live app: tool keywords and annotations, generation with a picture, lineage, design, readable errors, the app-not-running message); the UI-format-to-API workflow converter against seven official ComfyUI templates (ComfyUI 0.37, including the two Qwen-Image 2.1 ones, its promoted-widget subgraphs, dynamic combo and autogrow sockets), compared input for input with what the real frontend exports (subgraphs and promoted widgets, dynamic combos, autogrow inputs, bypass/mute, PrimitiveNode/Reroute) and UI exports imported through the API with and without ComfyUI; the six built-in templates that were converted this way (Flux schnell, Kontext edit, Wan 2.2 TI2V, ACE-Step song, Qwen-Image 2.1 txt2img and edit) passing server-side validation with their own defaults - including that a model file missing from /object_info reads as "download it", never a crash - and generating against the fake backend's real /object_info; the auto|qwen21|flux|sdxl image engine choice (installed-model detection, per-project default, per-call override, graceful fallback) and Qwen-Image 2.1's multi-reference edit (1-10 images, extra reference nodes wired or dropped to match); lyric timing from section tags, storyboard pools and cut-on-lyrics; a long transition render that must keep its full length; character-consistency routing and its consistent_needs_reference error; the finishing filter graphs (colour grade, grain, vignette, letterbox, glitch) snapshot-tested plus a real ffmpeg render; byte-identical reproduction through "reuse recipe"; @mention edge cases; checkpoint, sampler and missing-node validation; hostile workflow imports; import traversal, symlinks, renamed files and the allowed-folder rule; SPA and asset path traversal; the browser-attack guard; job restart recovery, VRAM waiting and cancellation; the beat tracker on click tracks, drum patterns and the demo song; auto-cut invariants; ASS escaping of hostile lyrics; real ffmpeg renders in a folder named like the Windows install (apostrophe, spaces, accents); design golden hashes, bleed and text fitting; thread safety of the store; and the Faustus manifest check.

npm run build in frontend/ passes with zero TypeScript errors. CI runs the tests on Ubuntu and Windows with Python 3.11, 3.12 and 3.13 and builds the interface with Node.js 22.

Characters that stay the same

The kit is how a character survives many shots, engines and projects (docs/CHARACTERS.md has the details):

1. Canonical - one good image of the character (Cast -> Edit).

2. Model sheet - Kit -> Model sheet renders the canonical from set views with the edit engine; each view lands in the dataset with a caption.

3. Dataset - add the best takes, fix captions (the trigger word stands for the permanent look; captions describe pose, framing, light), watch the readiness report.

4. Training - pick the architecture of the engine you render with and start; the adapter is installed in ComfyUI and attached. From then on every @Name render with that engine loads it, and productions render the lead's shots txt2img + adapter (free poses) instead of editing the canonical.

5. Takes - score identity, keep the good ones, reject the drifted ones, promote a better canonical.

6. Pack / library - export a .hoardchar or save a version to the library; cast it into any project, or use it as a recipe's lead (cast={"lead": "lib_..."}).

Training runs a separate trainer program. Configure it in Kit -> Training -> Configure (or training in data/backend.json):

"training": {
  "lora_dir": "D:/LocalAI/ComfyUI/models/loras",
  "gpu": "auto",
  "trainers": [{"kind": "ai_toolkit", "name": "ai-toolkit", "dir": "D:/LocalAI/ai-toolkit"}],
  "base_models": {"qwen_image": "Qwen/Qwen-Image", "flux1": "black-forest-labs/FLUX.1-dev"}
}

--demo configures a fake trainer so the whole flow can be tried without a GPU.

Roadmap / known limits

• The QA director's checks are heuristics (thresholds in settings.qa.thresholds); its bible/prompt/reference scores need a vision model behind Hoard Link, and picking the final take by ear is still yours.

• A recipe abstracts the lead's name, look, negative, palette and bio; shot prompts that describe the old lead's props ("the lantern head") are listed as warnings for you to rewrite, not rewritten.

• consistent=true keeps a character's design from its canonical reference; Kontext takes a single reference (Qwen-Image 2.1 up to 10) and Wan is image-to-video only.

• Lyric auto-timing is an estimate from the song's structure, not vocal alignment; re-time by ear in Audio > Lyrics timing, or import an LRC aligned elsewhere (the example used faster-whisper), for a final cut.

• Section labels are "section A/B" with an energy level, not verse/chorus; very fast songs (around 170 BPM) are reported at half time.

• The MCP tools poll jobs (studio_job can wait server-side); the family hub hears prospero.job.* events, other clients do not.

• Timeline editing is clip-level, Ken Burns is a zoom range plus a pan direction, and colour grades are filter approximations, not 3D LUTs.

• The QR layer of the designer draws a placeholder box.

• Graphic shots (kind: "graphic") are drawn with Pillow and bundled fonts: no complex-script shaping, the fonts are the bundled ones only, a graphic clip is exported to NLE XML/EDL as its poster still, beat effects do not touch it, a clip-mode graphic and an overlay both need a span on the song, kinetic lyrics need timed lyrics (or lines you write), and the alpha overlay is a working QuickTime file. QA checks their text boxes exactly (clipped, outside the safe area, over the caption band) without OCR, and that a second render is identical.

• A short's script is written by whatever local model Hoard Link finds: check its facts before publishing (settings.script_review pauses for that). Nothing is uploaded to the platforms; publish.txt is for pasting.

Studio and Spaces

The studio supports direct creation and Spaces · Create with nodes. The person + outfit → picture → clip starter accepts an optional motion guide. Upload references in their nodes, assign optional purposes, preview them at full size and reuse results. The searchable palette retains every node type and existing technique. Clip duration controls the real workflow; excess references and conflicting guides produce actionable errors. See the research and local evidence.

License

MIT - see LICENSE. Bundled fonts keep their own licences: SIL Open Font License (prosperos_hoard/fonts/*/OFL.txt), except Special Elite (Apache License 2.0, prosperos_hoard/fonts/SpecialElite/LICENSE.txt).

Shared services (HoardLink 0.8.1)

Downloads use Links through the family media client, and synthesis remains available through voice_tts. Existing local backend discovery is retained.

The vendored copy is maintained by HoardLink’s sync script. Windows validation and the family service contract are documented in HoardLink’s docs/commons/windows-validation.md and docs/commons/services.md.

Family expansion · 2026-10-04

studio_outpaint expands an existing image onto a larger canvas and creates a traced mask, then submits the existing inpaint job through the normal queue. The original asset remains unchanged; generated pixels, including VAE reconstruction, must be reviewed. Use a multiple-of-eight size no smaller than the source, up to 4096 px, and normalized anchors. The library lightbox offers +50% expansion after entering an edit prompt.

The seventh TTS adapter connects to an already-running local Yovoice audio.cpp service. Set YOVOICE_URL to a loopback HTTP origin with port, YOVOICE_MODEL to an installed model ID and YOVOICE_API_TOKEN (32+ characters) or YOVOICE_TOKEN_FILE. studio_voices checks real runtime/model readiness. Synthesis uses PUT job IDs, polling, cancellation and WAV validation. Options differ by model family; the server catalogue is authoritative. No model is downloaded automatically and CPU/Vulkan/CUDA support depends on that runtime and model. Model-weight licenses are separate. HTTP protocol tests do not prove audio quality on an installed model.