agent-video-engine

A local, headless video editor for AI agents. An agent controls source-linked cuts, captions, overlays, audio and renders through MCP tools or a JSON command line interface. There is no desktop editor or timeline UI.

The engine saves one editable project with revision history. The agent supplies editorial decisions; the engine validates edits, maps source timing to output frames and renders through Remotion.

Requirements and setup

• Node.js 22 or newer.

• Installed ffmpeg and ffprobe executables.

• An installed Chrome or Chromium executable for rendering.

• For transcription: installed whisper-cli from whisper.cpp and a cached compatible GGML model.

Install the pinned JavaScript dependencies and check the local tools:

npm ci
node src/cli.mjs doctor

The engine does not download a browser or transcription model at runtime. It discovers common local Chromium installations, or uses AGENT_VIDEO_BROWSER when set to an executable path. Transcription defaults to ~/.cache/whisper.cpp/ggml-medium.en.bin; set VIDEO_ENGINE_MODEL or supply a transcription request's modelPath to choose another installed model.

doctor reports transcription dependencies even when only rendering is needed. A missing model does not prevent creating projects, importing transcripts or rendering.

Related MCP server: YoloCut

MCP connection

Configure an MCP client to launch the stdio server directly:

{
  "mcpServers": {
    "agent-video-engine": {
      "command": "/absolute/path/to/node",
      "args": ["/absolute/path/agent-video-engine/src/server.mjs"]
    }
  }
}

Replace both paths with local installations. If needed, pass AGENT_VIDEO_BROWSER and VIDEO_ENGINE_MODEL through the client's environment configuration. The server reserves stdout for MCP protocol messages and starts on demand.

MCP hosts receive typed tool schemas. node src/cli.mjs tools also lists tool names and descriptions.

Command line usage

Run any tool with a JSON request file, or read the request from stdin:

node src/cli.mjs project_create create.json
node src/cli.mjs project_inspect - < inspect.json

Example create.json:

{
  "projectDir": "/absolute/path/video-project",
  "name": "Example edit",
  "width": 1080,
  "height": 1920,
  "fps": 30
}

Example inspect.json:

{
  "projectDir": "/absolute/path/video-project",
  "includeTimeline": true
}

New projects start at revision 0. Add a local source with media_add:

{
  "projectDir": "/absolute/path/video-project",
  "expectedRevision": 0,
  "path": "/absolute/path/media/source.mp4",
  "addToTimeline": true
}

The response includes the asset ID and new revision. Use those returned values in subsequent requests. Media is referenced by its canonical local path and fingerprint; it is not copied into the project.

Editing workflow

1. Inspect the project and current revision.

2. Add media and import a timestamped transcript, or start local transcription.

3. Read transcript words and inspect source previews before choosing cuts.

4. Apply edits using the current expectedRevision.

5. Run review_cuts, inspect the report and short join previews, then revise.

6. Validate the project and render a new final export.

Every successful edit batch creates a revision. Invalid batches and stale revisions leave the project unchanged. Restoring an earlier revision creates a new revision and preserves the intervening history.

Source files are never edited. Exports use new filenames and cannot replace an existing file or registered source. Built-in processing stays local and has no publishing or upload tools. Numerical checks establish timing and encoded properties; editorial quality and the final audiovisual result still need review.

Available tools

Preview kind is still or preview. Frame ranges use output frames; source preview bounds use source seconds. Transcription and rendering return durable job IDs. Follow job_get until completion, failure or cancellation. Completed stills can be returned as MCP images using includePreview.

Detached workers preserve jobs after a client disconnects. A local loopback socket serializes workers to limit concurrent resource use. Job metadata is saved under the project's .engine/jobs/; shared worker state defaults to the engine's .cache/, configurable with VIDEO_ENGINE_RUNTIME. Failed or interrupted jobs can be retried when their original project revision remains current.

Review speech cuts

review_cuts reviews discontinuous joins in the built-in timeline without changing its project or revision. Contiguous splits of the same source with unchanged playback and layout are omitted. It returns a durable job ID; use job_get to read the completed report and preview paths.

{
  "projectDir": "/absolute/path/video-project",
  "expectedRevision": 3,
  "contextSeconds": 1,
  "offset": 0,
  "limit": 20,
  "renderPreviews": true
}

The report identifies each join in output frames and seconds, and both sides in original source time. It flags boundaries inside timestamped words, large audio-level changes and long silence near a join. Audio measurements use the rendered master, including music, overlays, volume and playback rate. They are cues for review, not proof of a clipped syllable or an unnatural edit.

Previews include approximately contextSeconds on each side, rounded to whole frames and clamped to the composition bounds. The default is one second per side; supported values are 0.25 to 3 seconds. Reviews are paginated: offset starts at zero and limit defaults to 20, with a maximum of 50 joins per job. Continue with the returned nextOffset when hasMore is true. This bounds rendering work on long videos.

The default audio checks compare 150 ms RMS windows and flag level changes of at least 12 dB when both sides are above the -45 dBFS silence floor. Quiet gaps of at least 0.4 seconds are flagged. Stereo channels are measured by energy, preserving opposite-phase audio. Reports mark truncated measurement windows, silence that reaches a preview edge and neighboring cuts that limit attribution. These fixed heuristics describe the mixed master; they do not classify speech or intentional pauses.

Use renderPreviews: false for transcript timing review without rendering or audio analysis. Missing transcripts, absent audio and measurements limited by preview context are reported explicitly. Review files stay under the project's ignored .engine/reviews/ directory. Cancellation preserves the original edit and removes incomplete review output.

Custom compositions can ignore or change the engine's timeline, so they are rejected with UNSUPPORTED_CUSTOM_REVIEW rather than given misleading join timestamps. Review their rendered output directly, or use the built-in timeline.

Edit operations

edit_apply accepts a batch of up to 1,000 operations:

A main clip has id, assetId, sourceStart and sourceEnd. Optional playbackRate, volume and fit default to 1, 1 and "cover". fit can also be "contain". Optional position contains pixel values {x,y,width,height}.

An overlay has id, type, start and end. Its type is text, image, video or audio. Text overlays need text; other types need an appropriate assetId. Timed overlays support optional sourceStart and volume. The style object supports positioning, size, opacity, crop, typography and fadeInFrames/fadeOutFrames.

Output settings are width, height, integer fps from 1 to 120, and background. Caption fields are enabled, fontSize, color, highlightColor, bottom and wordsPerPage.

Example edit request:

{
  "projectDir": "/absolute/path/video-project",
  "expectedRevision": 3,
  "operations": [
    {"type": "delete_source_range", "assetId": "camera", "start": 2.1, "end": 3.4},
    {"type": "set_captions", "patch": {"enabled": true, "wordsPerPage": 4}},
    {"type": "upsert_overlay", "overlay": {
      "id": "callout", "type": "text", "text": "Look here", "start": 0, "end": 2,
      "anchor": {"assetId": "camera", "wordId": "w42", "offset": 0.1},
      "style": {"fontSize": 56, "top": 100, "height": 120, "fadeInFrames": 5}
    }}
  ]
}

delete_source_range affects every retained occurrence of that source interval. Use clip-specific trimming or removal to change one occurrence. Overlay positions and durations are not automatically inferred from spoken content.

Source timing and word anchors

The manifest stores source seconds without independently rounding every cut. Compilation rounds cumulative output boundaries onto one integer frame clock. This prevents accumulated duration drift across many cuts. Captions follow retained clips, including repeats, reordering and playback-rate changes.

Words retain stable IDs and original source times. Native point timestamps, where start equals end, compile as one-frame captions. Deleting a point-timed word raises a timing error because it has no measured speech interval.

An optional overlay anchor attaches it to a source word:

{"assetId": "camera", "wordId": "w42", "offset": 0.1, "clipId": "optional-occurrence-id"}

The stored end - start supplies duration. The compiled start follows the edited word plus offset, clamped to zero. Removing the speech suppresses its overlay. The first edited occurrence is selected by default; clipId selects a specific occurrence. Omit anchor for absolute output timing.

Overlay endpoints extend the composition when needed. Overlay-only titles, graphics and audio projects can have an empty main timeline.

Transcript and CutScript imports

transcript_import accepts word arrays, CutScript transcript JSON or native whisper.cpp JSON. Plain word entries use id (optional), text or word, start, end and optional confidence. Times must be finite, nonnegative and ordered in source time.

project_import_cutscript creates a new engine project from a version-1 .aive. It resolves a relative videoPath against the original project directory, preserves the full source transcript and keeps everything outside the union of explicit deletedRanges. Untranscribed lead-in, gaps and trailing footage survive unless explicitly cut. The original .aive and media remain unchanged.

Local transcription retains raw output and normalized words under .engine/transcripts/. Temporary extracted audio is removed. Small decoder overshoots can be bounded to the actual media duration and marked for timing review, with raw timings retained.

Custom Remotion compositions

Use custom React/Remotion code for original motion, demonstrations or more specialized visual behavior. The entry point must call registerRoot():

{
  "type": "set_custom",
  "custom": {
    "entryPoint": "/absolute/path/composition/src/index.ts",
    "compositionId": "ExampleComposition",
    "publicDir": "/absolute/path/composition/public",
    "props": {}
  }
}

Custom compositions receive {...custom.props, timeline}. The timeline contains compiled clips, captions, overlays and registered asset src URLs. Consume that timeline when engine edits should affect the composition. Existing code with hardcoded timing retains its behavior until adapted.

custom.publicDir is optional and must identify an absolute local directory. staticFile() and getStaticFiles() use token-protected loopback routes for inventoried files, including symlinked files and directories, without copying those media files. Unregistered local paths are not served. Custom composition code is trusted executable project code.

The render-only bridge pins Remotion and @remotion/renderer to 4.0.534. It uses esbuild and a compatibility layer adapted from upstream render internals, without Studio, the Remotion CLI, Webpack or Rspack. A mismatched runtime or composition dependency declaration fails explicitly. Upgrade Remotion packages together and verify the bridge before adopting another version. Consult the repository's third-party notices for attribution and dependency licensing.

Development checks

npm test
npm run check

npm test runs project, timing, migration and boundary checks. npm run check also enables real headless synthetic rendering and an MCP client workflow. Tests cover cuts, captions, word anchors, revisions, source previews, encoded frame counts and audio, custom motion, symlinked public media, cancellation and source preservation. Test media is generated in temporary directories.

Keep personal footage, projects, caches, models, credentials and generated exports outside the repository. The included ignore rules are a precaution; inspect staged files before committing.