Allows a local LLM agent to drive Unreal Engine 5 scenes through verified macro tools, including applying standing animation clips, measuring character poses and foot placement, capturing evidence frames, searching/previewing clips, and managing bounded workflows for crowd loops. Future roadmap includes building environments, pathfinding and collision-aware locomotion, multi-character staging, camera framing, and recording reference videos.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@unreal-local-director make the audience chat quietly"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
⚠️ Discontinued
This project (now called "Antigo Director", the Old Director) is no longer being developed. Its successor is Director. The code and the published versions stay here for reference. An explanation of why will come later.
Turns human intent into verified actions in Unreal Engine, using a local LLM that, on its own, could not operate the engine reliably.
This is not "an AI tool for Unreal". The model is the thin part: it reads a plain-language request and picks from a catalogue of ready-made, measured movements and macros. The thick part is everything around it: the tools that measure the result (pose, feet, bones, collisions), refuse what they cannot do, restore the scene, and notice when the agent is stuck. The purpose is reference videos (blocking, character motion, camera) for video models such as MiniMax H3 and LTX 2.5, without paying a frontier model for every click.
v0.0.6, early: a corrective version. An external read-only audit (Codex "Astra", 10/10/2026) found 29 problems in the executor, the ruler and the night-campaign code; v0.0.6 fixes them, each with a test that fails on the previous code. The fixes themselves have not been re-audited. Published so others can follow the approach and the evidence. Earlier snapshots:
v0.0.1,v0.0.2-wip,v0.0.3,v0.0.4-wip.
The end state is directing, not operating. You say what you want, in plain words, and the scene happens:
"Make X do this" → the character does it: pick the motion, apply it, check the pose, show you a frame.
"Build a little square where X can stroll with Y" → the agent builds the square (layout, ground, benches, trees, lighting, from ready-made assets), places both characters, makes them walk together without colliding, frames the camera and records the reference clip.
Today we are only at the first rung: verified standing animations on existing characters. Building places, walking paths and multi-character staging are the future and most important part of the roadmap.
The aim is intuitive and evolutive, not pretty. Reference videos only need believable blocking, motion and interaction, and that means physics: gravity, collisions, characters bumping into furniture, objects that fall, get pushed or carried. Unreal gives this out of the box (Chaos physics, character movement, collision channels, navigation). Blender is easier and more mature to drive from an agent (its Python API is excellent, and the Blender MCP projects we learned from are ahead of anything for Unreal), but physics and interaction between characters and objects are hard there. We chose the harder-to-drive engine because it gets the important part right, and we are building the missing agent layer ourselves.
Frontier models (Claude Opus, GPT/Codex-class) can already operate Unreal reasonably well. They are also slow to iterate with and expensive per action. This project is the opposite bet: a small, cheap, local model plus a thick layer of deterministic tools that do the hard part, measure the result and refuse to lie.
Related MCP server: unreal-mcp
Unreal 5.8 ships an official MCP server with 29 toolsets and ~593 atomic tools (get/set properties, spawn actors, capture viewport, run editor scripts...). A frontier model can compose those. A local model could not:
What we measured (859 real calls by our local agent) | Why it happens |
~15% of calls failed | guessed property names ( |
| the model writes editor Python on the fly, with the wrong API |
"Done!" with the character lying on the floor | Unreal's bounding box does not follow the animated pose: a character lying down still "measures" 180 cm tall |
Long hangs, editor modal dialogs, Play-In-Editor | one bad call blocks everything; the model has no way to know the state |
Atomic tools give the model power without verification. A small model needs the opposite: few tools, each a whole task, each one measuring its own result.
Borrowed from Blender agent projects (see Credits), re-engineered for Unreal:
Layer | What it is | Who uses it |
Atomic | the ~593 official MCP tools | only our macros, internally |
Macro | one sentence = one tool: | the local LLM |
Workflow | bounded processes: | the local LLM |
Truth | deterministic gates: standing, feet on the floor, actor didn't move, no neighbour collision, legs planted | every macro |
Vision | second opinion on a captured frame — never approves alone | workflows |
Key pieces (all in src/unreal_macros/):
The "parallel studio" (idea of the project owner): to measure a pose we copy the character into a neutral, closed grey room far from the scene, capture with and without the copy, and the pixel difference is the silhouette. A projected height ruler converts pixels to cm. The original is never touched. Studio v1 used the sky as background (cyan sky ≈ cyan mannequin, and it over-exposed); v2 is a closed unlit grey room with fixed exposure, and the measuring copy is painted unlit white:
v1 (sky) | v1 after editor changes | v2 (grey room) |
|
|
|
Gates: silhueta_de_pe (160–210 cm), pes_no_chao (±4 cm to the real floor), pose_em_todo_clipe (4 instants of the clip), pes_plantados (leg-mask IoU: walking scores 0.0, talking 0.7–0.8), ator_parado, distancia_vizinhos, sem_oclusao_cenario.
Every failure restores the previous state and checks the restore. A single lock: one operator in Unreal at a time.
Escape hatch chamada_avancada: read-only access to the atomic tools, logged.
Clip library search (buscar_clipe, previa_clipe): text search + 3-frame previews over a local Mixamo-style library (not included).
Contract tests (tests/testes_contrato.py): 22 cases including negative controls (walking, falling, kneeling, missing roll fix, glued neighbour, forbidden saves) that must fail.
Sandbox guard (sandbox_guard.py): lets an agent experiment with code changes against Unreal while refusing saves, deletes, level changes, editor scripts and any non-test actor.
Docs (Portuguese, as written during the work): docs/pt/ — tool policy, macro list, MCP inventory, the overnight agent benchmark analysis and an external review.
The runtime target is Qwen3.8-Flash-Next (UD-IQ4_XS quant) served by Strata on a single 48 GB GPU machine, driven by the Hermes agent. In our experience it is far better suited to this job than the dense Qwen3.8 27B, which was slow and quite limited for tool-heavy Unreal work: long waits per step and many more malformed calls.
An overnight benchmark (55+ attempts, every claim cross-checked against the real tool reports) showed the local agent is honest (222/224 numbers it logged matched the tool output) and good at finding bugs in our macros — but weak at obeying "stop" / "keep going" orders. Details: docs/pt/ANALISE_BANCADA_2026-10-06.md.
Frontier models were used to build and review this scaffolding, not to run it.
Works in our lab, with measurements:
standing gestures and talk loops on existing characters, with verified pose and feet (macros, v0.0.1);
chained movements in a Sequencer timeline: stand up, walk a distance, push a door open and walk through (v0.0.3; see below);
an in-editor Python toolset (DirectorTools) for what the official MCP cannot do: full actor inventory, bone positions, isolated capture of one actor;
a night of attempts by the local model that leaves the scene as it found it (v0.0.6; see below): 28 real attempts in 7 hours, scene restored after every one, no program failure.
Missing (and measured as missing): turning toward a target, sitting on an existing seat, picking up objects, waiting for another character, stairs. Lighting of the test captures is dark. Distance control is quantized by the walk cycle. The local model still does not carry a whole night to the deadline on its own (it closed the campaign early), and nothing here has been tried on a second character mesh.
No new features for the scene. This version makes the night campaign safe and honest: what the executor deletes, how attempts are reserved, what counts as success, how the ruler measures, how the supervisor ends a night, and what evidence stays on disk. It answers the 29 findings of the external audit of 10/10 plus the items inherited from v0.0.5, in 7 batches. Every fix has a test that reproduces the defect on the previous code and passes after.
Batch | What changed | New tests (all pass now; most were red on the previous code) |
Scene | the executor only deletes what is provably its own (a name prefix is not proof); the scene check compares path, label, class, position, rotation and scale; cleanup only counts with a positive confirmation; before/after inventories kept | 12 (11 red before) |
Reservations and lock | one admission at a time per round; a lost state file never becomes a new round; the lock is only released by the process that took it | 24 |
Success | "passed" means measured, nothing failed afterwards and the scene restored; one single rule decides success (3 distinct attempts, same conditions and versions) | 13 |
Ruler | no limit or quota changed; non-finite coordinates invalidate the measurement; transitions are judged on full values, not rounded ones; a foot that starts in the air is not a lift-off | 16 |
Supervision | the watcher only accepts the ending of its own campaign; dry-run writes nothing; an orphan lock is released only when proven orphan | 17 |
Evidence | samples of every attempt on disk; a corrupted history blocks but does not vanish from the report; a video only exists complete | 15 |
Gate | 12 suites that were outside the battery are now in it | — |
Measured:
Battery (tools/bateria.py): 34 offline suites, 420 cases, 13 Hypothesis properties, no new lint or type errors, all green.
One real attempt in Unreal 5.8.3 (11/10): score 0, approved; scene identical before and after (86 → 86 actors, compared on path, label, class, position, rotation and scale); the 4 cleanup steps confirmed; lock free at the end.
First real night (11→12/10, the local model driving, 7 h 04 min): 28 attempts, scene restored in 28, 0 program failures; 11 approved with score 0, 17 rejected by the ruler. The model closed the campaign 25 minutes before the deadline, declaring there was nothing left to try.
Library (biblioteca/): 173 → 423 measured clip transitions. The ruler changed version, so the one validated block answers "measure again" until it gets 3 new approved attempts.
Added on 13/10, after that night: "test the version" mode. The list of what the version changed and can be exercised live lives next to the code (tools/testes_da_versao.json, 6 tests today). The program, not the model, checks the campaign records to say what was covered, and refuses to close a campaign as "exhausted" while a test is missing. 16 offline tests (13 red on the previous code). This mode has not run live yet.
Also first published here (written during v0.0.5): the night-campaign layer (tools/director_noite.py, src/unreal_macros/campanha.py), macros that work without a connection, and a process check that tells alive / dead / unknown apart. v0.0.5 was never published on its own; its aggregate results are in docs/pt/v005_resultados/.
Not proven: long autonomy of the local model; the test-the-version mode running live; a real video made with this version; other character meshes. Aggregate numbers: docs/pt/v006_resultados/; full list of changes: CHANGELOG.md.
The local model chooses clips and recipes; deterministic, tested code does the measuring, logging and protecting.
Ruler and recipe (src/unreal_macros/fila.py, receita.py): criteria for stand up, wave, walk, stop, transitions and duration; per-step trims (inicio_q, corte_fim_q) and end turn (turn_fim_deg).
Single queue executor (tools/fila_executor.py): lock, assembly, measurement, logging with the measurement conditions, cleanup and resume. An external review (Codex, read-only) found 3 holes; all fixed with tests.
Library (biblioteca/): 138 measured clip transitions; the same transition costs the same in any recipe.
Break battery (tools/bateria.py): lint and type check against a baseline, 16 offline suites and Hypothesis properties in a clean worktree (~40 s, all green).
Four work lines, one command each (tools/director.py): idea scouting by the local model, the battery, the big Unreal test (a scene with variations, each with video, the operator picks one) and the external review.
Local dashboard (tools/painel.py): run buttons, live history, score chart, videos, picking a variation, approving ideas; each run ends with a short program-built summary.
Not proven yet: the big Unreal test (scene with variations) has not run live; the version is not closed.
Decision: the local model does not invent motion. It chooses and chains ready-made movements whose numbers were measured on the clips themselves (src/unreal_macros/blocos.py): andar(distance), levantar(), abrir_porta(gap, hinge). The DirectorTools toolset builds them as a Level Sequence (seq_build), so the result renders the same way every time and becomes video.
Vertical test ladder (from "Marcos stands up" to "Marcos and Afonso sit and drink tea"; gap map):
Request | Result | What was measured |
V01 "Marcos stands up" | 5/5 | starts seated, feet never inside the seat, ends standing (hips 94 cm), toes on the floor |
V02 "… and goes to the door" | 4/4 | 2 cm hip jump between clips, never crosses the door, stops 54 cm from it (target 60) |
V03 "… goes to the door and opens it" | 4/4 | door opens 77°, starts moving when the right hand touches it, walks through, no bone inside the walls |
Videos: walk · V01 stand up · V02 stand up and go to the door · V03 open the door (test captures: only the listed actors are rendered, dark background).
The door is pushed by the body, not animated by hand: in every frame the leaf opens just enough for none of 15 bones to cross it, and never closes back (blocos.angulos_porta). That is how we found a real limit: the Mixamo push-door clip throws the left arm wide, so with a 100–120 cm doorway the arm went 10–15 cm into the jamb. The block now carries a measured precondition (gap ≥ 150 cm, hinge on the pusher's left) and refuses outside it, instead of producing a wrong scene.
Run by the local model (Hermes on Strata/Qwen3.8-Flash-Next, 06/10 night): 27 requests from a non-expert, using only the v0.0.1 production tools. Self-reported score, not yet cross-checked against the macro reports: 9 done with measurements, 17 refused with the right reason, 1 failed with a measured cause, 0 stuck, scene restored exactly. For the refused ones it wrote plans in ready-made movements and listed what is missing; its top gap is virar_para (turn toward a target), the silent prerequisite of walking. Raw data: docs/pt/hermes_v003/.
Also in v0.0.3:
DirectorTools (editor_python/): installed through UE_PYTHONPATH, no change to the game project; it can reload itself.
Isolated capture adapted from VERA (MIT, credited in THIRD_PARTY_NOTICES.md).
Pitfalls of the native UE 5.8 MCP checked live against ue58-mcp-field-notes: the 20-actor cut does not happen on 5.8.3, optional arguments are required, and Sequencer poses need force_evaluate.
Next, v0.0.4:
external audit (Codex "Astra");
the gaps Hermes found: turn, sit on an existing seat, pick up, wait for another character;
lighting.
Theme: the agent notices it is stuck, knows what to do, and does not waste hours. Report ids, lock with a verifiable owner, experiment controller, stuck detector, lessons, requests bank, value queue, supervisor, session hand-off, rotation. Offline suite 137/137; the real-Unreal validation was stopped midway and is being paid back selectively during v0.0.3 (status, changelog).
Unreal Engine 5.8 with the official MCP plugin listening on http://127.0.0.1:8001/mcp
Python 3.11, pip install -r requirements.txt
A level with Y Bot–style characters; a Mixamo-style FBX clip library (set ULD_CLIP_LIBRARY)
ULD_EVIDENCE_DIR for captured frames
Run the MCP server: python src/server.py (stdio), then point your agent (Hermes, or any MCP client) at it.
v0.0.3 movements: install DirectorTools (editor_python/LEIA.md), then python tools/bloco2_v.py V01|V02|V03.
Issues and ideas welcome — especially: locomotion with collision gates, bone-based pose measurement, faster/stable capture, English tool names. A new macro is only accepted with 3/3 positive cases and a negative control that fails (policy rule 11).
Who | Role |
Talis | project owner, direction, the "parallel studio" idea, lab |
Claude Code (Anthropic) | implementation, tests, measurement in Unreal |
ChatGPT (OpenAI) | architecture and planning reviews: the "Pedidos Atendidos" metric, the vertical test ladder, the v0.0.3 plan v2 (freeze v0.0.2, toolset first, gap-driven blocks) |
Codex "Astra" (OpenAI) | external code review and audit: the v0.0.4 review and the audit of 10/10/2026 (29 findings) that v0.0.6 answers |
Hermes on Strata / Qwen3.8-Flash-Next | the runtime agent: runs the requests, the test benches and the night campaigns; as "junior" it also scouts the code for problems (three of its findings, N1, N3 and N4, were fixed in v0.0.6) |
AI contributors do not have GitHub accounts, so they are credited here and in NOTICE.md rather than in GitHub's contributor graph.
Project | What we use |
ezesubu/VERA — Virtual Engine Reasoning Agent (MIT, © 2026 EazyLabs / maVERAick) | code adapted: isolated actor capture in |
oliver-io/unreal-harness (MIT, © 2026 Oliver Carrillo) | ideas: gates, dry-run, closed error taxonomy, progressive disclosure, real-editor tests |
Details and the exact files: THIRD_PARTY_NOTICES.md and NOTICE.md.
Custom, source-available, not open source — see LICENSE. This project draws on ideas from other projects and we have not yet cleared what can be fully opened.
O Director transforma intenção humana em ações verificadas no Unreal, usando uma LLM local que, sozinha, não teria capacidade de operar o engine de forma confiável. Não é uma "ferramenta de IA":
O modelo só escolhe: entende o pedido e escolhe entre movimentos prontos e medidos.
O valor está em volta dele: medir o resultado, recusar o que não sabe fazer, devolver a cena e perceber quando o agente emperrou.
O objetivo é gerar vídeos de referência para MiniMax H3 e LTX 2.5.
Na v0.0.3:
Escada vertical (docs/pt/MAPA_LACUNAS_V.md):
"Marcos se levanta" 5/5;
"… e vai até a porta" 4/4;
"… e a abre" 4/4, com a porta empurrada pelo corpo e uma pré-condição medida;
tudo montado no Sequencer, com vídeo.
Hermes (Strata/Qwen) no teste de Pedidos Atendidos: 9 atendidos, 17 recusados com o motivo certo, 1 falha justificada e 0 emperramentos. Os números são dele e ainda vão ser conferidos.
Na v0.0.6 (corretiva, 13/10/2026):
corrige os 29 apontamentos da auditoria externa (Codex "Astra") de 10/10, cada um com um teste que falha no código anterior;
bateria: 34 suítes, 420 casos, tudo verde;
primeira noite real com o modelo local: 28 tentativas, cena restaurada nas 28, nenhuma falha do programa, 11 aprovadas com nota 0;
o modelo fechou a noite 25 minutos antes do prazo; por isso entrou o modo "testar a versão", que ainda não rodou ao vivo.
Por que Unreal: física fácil e interação entre personagens e objetos, que o Blender não tem pronta. Documentação de trabalho em docs/pt/.
This server cannot be deployed
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to control Unreal E…
Control Unreal Engine to browse assets, import content, and manage levels and sequences. Automate…
Build editable 3D scenes, direct characters and cameras, and export AI video references with MCP.
Video, image and audio generation on top AI models for agents, with an AI director built in.
