shotplan

An MCP server for turning landscape video into vertical reels — where the model can look at what it is about to do before committing to a four-minute render.

Content-aware reframing runs entirely on your machine: spectral-residual saliency plus on-screen text detection build a per-frame importance map, and a smoothed virtual camera holds the important region so UI text stays legible at 9:16.

This is deliberately not an ffmpeg wrapper

There are already a dozen ffmpeg MCP servers. The largest exposes 161 tools and has a couple hundred stars, and that tool count is the diagnosis: transliterating a CLI into JSON Schema produces something strictly worse than letting the model call ffmpeg in a shell. The model already knows ffmpeg. A schema is a lossy re-encoding of knowledge it has, plus a permanent context cost on every session that loads it.

What MCP adds here is a loop the model can actually participate in:

analyze_shots  →  look at the contact sheet  →  preview_shots  →  adjust  →  render_reel
   (seconds)         (the model sees frames)      (seconds)                   (minutes, once)

preview_shots is the load-bearing tool. It renders the proposed treatment for specific shots as small stills in about a second, which turns each iteration from four minutes into three. Without it the loop is technically expressible and practically unusable.

Related MCP server: YOLO-FFMPEG-MCP

Install

claude mcp add shotplan -- uvx shotplan-mcp

Or by hand:

{
  "mcpServers": {
    "shotplan": {
      "command": "uvx",
      "args": ["shotplan-mcp"],
      "env": { "SHOTPLAN_WORKSPACE": "~/Movies/shotplan" }
    }
  }
}

ffmpeg must be on PATH, or install shotplan-mcp[bundled-ffmpeg] to get a wheel-bundled build. Two optional extras add capabilities: [ocr] for text-aware framing (also needs the tesseract binary) and [captions] for Whisper burn-in. Call doctor first — it reports exactly what is present, what each missing piece would unlock, and how to install it. Nothing fails silently and nothing fails late.

Tools

analyze_shots returns warnings per shot, not just numbers, because a JSON Schema cannot express "extend will streak on this shot because content reaches the frame edge." That judgement is the difference between output that looks intentional and output that looks cheap, so it ships as a resource (shotplan://guide/treatments) and as per-shot warnings on the specific shot where it applies.

Long-running work: why there's a job store

This is the part of building an MCP server that isn't obvious, so here is the reasoning explicitly.

Progress notifications can't be load-bearing. notifications/progress only fires if the client attached a progressToken to the request. If it didn't, progress is a silent no-op. It is a liveness mechanism, never a correctness one.

Client timeouts are the real constraint, and there is more than one. Claude Code's per-server timeout is a hard wall-clock ceiling that progress notifications explicitly do not extend. On HTTP/SSE transports there is also a 60-second timer to the server's first response byte. And on stdio the binding limit is the idle timeout — 30 minutes of silence aborts the call, while hours of chatter is fine. Other clients are reported to drop results far sooner (Claude Desktop at around 60s, per community bug reports rather than documentation). Timeout behaviour varies per client, often isn't configurable by your user, and your server cannot discover it — so don't design against a number you can't see.

The Tasks extension isn't ready to be the only path. It lands formally in the 2026-07-28 spec, the earlier 2025-11-25 shape is already dead, and client support in the wild is effectively nil.

So the server owns job state itself. render_reel returns a handle immediately and accepts wait: {max_seconds: N} — if the work settles inside N (default 20s, capped at 45s to stay under every client's timeout) the completed result comes back inline, so short clips feel synchronous. Otherwise you poll get_job. State persists to $SHOTPLAN_WORKSPACE/.shotplan/jobs/, so jobs survive a restart and a Tasks adapter can later be a thin shim over the same store rather than a rewrite.

One honesty note: the renderer does not emit frame-level progress, so percent is an elapsed-versus-estimate figure and says so in its message. It is not measured, and it is not presented as though it were.

The full reasoning, with sources and the exact timeout numbers, is written up in docs/long-running-mcp.md — including the payload-size traps (maxResultSizeChars is text-only; tool descriptions truncate at 2KB) that the quickstarts don't cover.

Other design decisions

Videos are never inlined. A 30-second 1080×1920 clip is 5–20 MB, which is 7–27 million base64 characters; Claude Code truncates MCP output at 25,000 tokens. Videos come back as file paths with structured metadata. The one place inline images are spent is the contact sheet, capped at 9 tiles, 320px each, JPEG q70 — past that the server returns a note telling the model to use read_frame instead, which is both cheaper and a better debugging idiom.

File access is sandboxed to the client's roots. The server calls roots/list and refuses paths outside those directories plus SHOTPLAN_WORKSPACE. Most MCP servers will read anything their user can read; honouring roots costs very little.

Renders run in a subprocess. So a cancel can kill the process group, an OpenCV or ffmpeg crash can't take the server down, and ffmpeg's stderr is captured as a real log.

Sampling is not used. Support is patchy and it is deprecated in the 2026-07-28 spec — and the architecture is inverted anyway. The server returns metrics and pictures; the client's model decides. Reasoning belongs where the reasoning lives.

Environment

Development

pip install -e '.[dev]'
pytest tests/ -v

The tests drive the server over stdio as a real client — handshake, tool surface, resource reads, sandbox refusal, and a full analyse-then-render round trip. A unit test on a Python function proves nothing about whether a tool is callable or whether its result survives serialisation, which is where MCP servers actually break.

Limits

• Tested on macOS and Linux. Windows is best-effort.

• Tuned for 16:9 → 9:16. Other source aspects work but the treatment classifier isn't tuned for them.

• anim and reveal treatments render but preview as a plain slice; full keyframe preview is planned.

• The extend classifier is heuristic. Check the contact sheet rather than trusting it.

License

Apache-2.0. The reframing engine lives in reel-maker.