# shotplan

[![CI](https://github.com/lrvaka/shotplan/actions/workflows/ci.yml/badge.svg)](https://github.com/lrvaka/shotplan/actions/workflows/ci.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)

An MCP server for turning landscape video into vertical reels — where the model can **look at
what it is about to do** before committing to a four-minute render.

<!-- mcp-name: io.github.lrvaka/shotplan -->

Content-aware reframing runs entirely on your machine: spectral-residual saliency plus
on-screen text detection build a per-frame importance map, and a smoothed virtual camera
holds the important region so UI text stays legible at 9:16.

## This is deliberately not an ffmpeg wrapper

There are already a dozen ffmpeg MCP servers. The largest exposes 161 tools and has a couple
hundred stars, and that tool count *is* the diagnosis: transliterating a CLI into JSON Schema
produces something strictly worse than letting the model call `ffmpeg` in a shell. The model
already knows ffmpeg. A schema is a lossy re-encoding of knowledge it has, plus a permanent
context cost on every session that loads it.

What MCP adds here is a loop the model can actually participate in:

```
analyze_shots  →  look at the contact sheet  →  preview_shots  →  adjust  →  render_reel
   (seconds)         (the model sees frames)      (seconds)                   (minutes, once)
```

`preview_shots` is the load-bearing tool. It renders the proposed treatment for specific shots
as small stills in about a second, which turns each iteration from four minutes into three.
Without it the loop is technically expressible and practically unusable.

## Install

```bash
claude mcp add shotplan -- uvx shotplan-mcp
```

Or by hand:

```json
{
  "mcpServers": {
    "shotplan": {
      "command": "uvx",
      "args": ["shotplan-mcp"],
      "env": { "SHOTPLAN_WORKSPACE": "~/Movies/shotplan" }
    }
  }
}
```

`ffmpeg` must be on `PATH`, or install `shotplan-mcp[bundled-ffmpeg]` to get a wheel-bundled
build. Two optional extras add capabilities: `[ocr]` for text-aware framing (also needs the
`tesseract` binary) and `[captions]` for Whisper burn-in. **Call `doctor` first** — it reports
exactly what is present, what each missing piece would unlock, and how to install it. Nothing
fails silently and nothing fails late.

## Tools

| Tool | Cost | What it's for |
|---|---|---|
| `doctor` | instant | What's installed, what's missing, what that costs you. |
| `probe_video` | instant | Dimensions, fps, duration, and a render-time estimate. |
| `analyze_shots` | seconds | Scene cuts plus a per-shot plan: suggested treatment, raw metrics, **warnings**, and a labelled contact sheet. |
| `preview_shots` | ~1s | The proposed treatment for chosen shots, as stills. |
| `read_frame` | instant | One frame, when a contact sheet is too coarse. |
| `render_reel` | minutes | The full render. Returns a job handle. |
| `get_job` / `cancel_job` | instant | Poll or kill a render. |
| `burn_captions` | minutes | Whisper transcription burnt in. Run after reframing. |

`analyze_shots` returns `warnings` per shot, not just numbers, because a JSON Schema cannot
express *"`extend` will streak on this shot because content reaches the frame edge."* That
judgement is the difference between output that looks intentional and output that looks cheap,
so it ships as a resource (`shotplan://guide/treatments`) and as per-shot warnings on the
specific shot where it applies.

## Long-running work: why there's a job store

This is the part of building an MCP server that isn't obvious, so here is the reasoning
explicitly.

**Progress notifications can't be load-bearing.** `notifications/progress` only fires if the
client attached a `progressToken` to the request. If it didn't, progress is a silent no-op. It
is a liveness mechanism, never a correctness one.

**Client timeouts are the real constraint, and there is more than one.** Claude Code's
per-server `timeout` is a hard wall-clock ceiling that progress notifications explicitly do
*not* extend. On HTTP/SSE transports there is also a 60-second timer to the server's first
response byte. And on stdio the binding limit is the *idle* timeout — 30 minutes of silence
aborts the call, while hours of chatter is fine. Other clients are reported to drop results
far sooner (Claude Desktop at around 60s, per community bug reports rather than
documentation). Timeout behaviour varies per client, often isn't configurable by your user,
and your server cannot discover it — so don't design against a number you can't see.

**The Tasks extension isn't ready to be the only path.** It lands formally in the 2026-07-28
spec, the earlier 2025-11-25 shape is already dead, and client support in the wild is
effectively nil.

So the server owns job state itself. `render_reel` returns a handle immediately and accepts
`wait: {max_seconds: N}` — if the work settles inside N (default 20s, capped at 45s to stay
under every client's timeout) the completed result comes back inline, so short clips feel
synchronous. Otherwise you poll `get_job`. State persists to
`$SHOTPLAN_WORKSPACE/.shotplan/jobs/`, so jobs survive a restart and a Tasks adapter can later
be a thin shim over the same store rather than a rewrite.

One honesty note: the renderer does not emit frame-level progress, so `percent` is an
elapsed-versus-estimate figure and says so in its `message`. It is not measured, and it is not
presented as though it were.

**The full reasoning, with sources and the exact timeout numbers, is written up in
[docs/long-running-mcp.md](docs/long-running-mcp.md)** — including the payload-size traps
(`maxResultSizeChars` is text-only; tool descriptions truncate at 2KB) that the quickstarts
don't cover.

## Other design decisions

**Videos are never inlined.** A 30-second 1080×1920 clip is 5–20 MB, which is 7–27 million
base64 characters; Claude Code truncates MCP output at 25,000 tokens. Videos come back as file
paths with structured metadata. The one place inline images are spent is the contact sheet,
capped at 9 tiles, 320px each, JPEG q70 — past that the server returns a note telling the
model to use `read_frame` instead, which is both cheaper and a better debugging idiom.

**File access is sandboxed to the client's roots.** The server calls `roots/list` and refuses
paths outside those directories plus `SHOTPLAN_WORKSPACE`. Most MCP servers will read anything
their user can read; honouring roots costs very little.

**Renders run in a subprocess.** So a cancel can kill the process group, an OpenCV or ffmpeg
crash can't take the server down, and ffmpeg's stderr is captured as a real log.

**Sampling is not used.** Support is patchy and it is deprecated in the 2026-07-28 spec — and
the architecture is inverted anyway. The server returns metrics and pictures; the client's
model decides. Reasoning belongs where the reasoning lives.

## Environment

| Variable | Purpose |
|---|---|
| `SHOTPLAN_WORKSPACE` | Where outputs and job state go. Defaults to the working directory. |
| `SHOTPLAN_ALLOW` | Extra allowed directories, `os.pathsep`-separated. |
| `SHOTPLAN_FFMPEG` | Override the ffmpeg binary. |

## Development

```bash
pip install -e '.[dev]'
pytest tests/ -v
```

The tests drive the server over stdio as a real client — handshake, tool surface, resource
reads, sandbox refusal, and a full analyse-then-render round trip. A unit test on a Python
function proves nothing about whether a tool is callable or whether its result survives
serialisation, which is where MCP servers actually break.

## Limits

- Tested on macOS and Linux. Windows is best-effort.
- Tuned for 16:9 → 9:16. Other source aspects work but the treatment classifier isn't tuned
  for them.
- `anim` and `reveal` treatments render but preview as a plain slice; full keyframe preview is
  planned.
- The `extend` classifier is heuristic. Check the contact sheet rather than trusting it.

## License

Apache-2.0. The reframing engine lives in [reel-maker](https://github.com/lrvaka/reel-maker).