# framedeck

**Coding agents can't watch video. This makes video legible to them.**

Point framedeck at a TikTok, Reel, Short, or local file. It downloads the
video, works out which frames are actually *different*, writes them to disk,
and hands back a manifest of file paths the agent reads as images.

```bash
framedeck extract "https://vt.tiktok.com/ZSVKJNPcM/" -o ./out -n 10
```

```
tikwm  ·  editmateproject  ·  36.0s
  Cover undangan nggak harus foto prewed 👀 Foto masa kecil kalian juga bisa …

10 distinct frames → ./out
  [first ] t=   0.20s  frame-01-t0000.20.jpg
  [scene ] t=   6.50s  frame-02-t0006.50.jpg
  [scene ] t=  11.27s  frame-03-t0011.27.jpg
  [scene ] t=  13.07s  frame-04-t0013.07.jpg
  …
contact sheet: ./out/contact-sheet.jpg
manifest:      ./out/manifest.json
```

14 seconds end to end for that clip.

## How it works

![framedeck pipeline](docs/pipeline.svg)

<sub>Source: [`docs/pipeline.mmd`](docs/pipeline.mmd) — Mermaid, rendered with [line9](https://line9.ai): `line9 render docs/pipeline.mmd --out docs/pipeline.svg --theme blueprint`</sub>

## Why not just yt-dlp and ffmpeg

You can absolutely `yt-dlp | ffmpeg -vf fps=1/2` yourself. Three things go
wrong when you do, and framedeck exists for exactly those three:

**1. Uniform sampling is the wrong sampler.** A 36-second scroll-through of a
UI holds maybe nine distinct screens. Sampling every two seconds gives you
eighteen frames, half of them mid-scroll blur, and still misses the screen that
was only on camera for 800 ms. framedeck lets ffmpeg's scene filter nominate
candidates, adds uniform anchors so a slow continuous scroll isn't skipped
entirely, then drops near-duplicates by perceptual hash. You read ten frames
instead of eighteen and see *more*.

**2. Social extractors break.** yt-dlp's TikTok path was returning
`Unable to extract universal data for rehydration` the day this was written —
including on nightly. framedeck tries providers in order (local file → yt-dlp →
tikwm → oembed cover) and only fails when all of them do. When the video itself
is unreachable it still returns the caption and cover image, which is often
enough to answer the question.

**3. The output isn't shaped for an agent.** framedeck writes a `manifest.json`
of absolute paths, timestamps, hashes, and *why each frame was kept* — plus one
contact sheet for a cheap overview before reading frames individually.

## Install

```bash
pip install framedeck          # or: git clone && pip install -e .
```

Needs `ffmpeg` and `ffprobe` on `PATH`. `yt-dlp` is optional but widens the
supported sites considerably.

No Python dependencies. Not "few" — zero. The perceptual hashing runs on a 9×8
grayscale buffer that ffmpeg produces and Python bit-twiddles, so there is no
Pillow, no numpy, and nothing to conflict with in an agent sandbox.

<details>
<summary>Getting ffmpeg without root</summary>

```bash
mkdir -p ~/.local/bin && cd /tmp
curl -sSL -o ff.tar.xz https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
tar xf ff.tar.xz && cp ffmpeg-*-static/{ffmpeg,ffprobe} ~/.local/bin/
curl -sSL -o ~/.local/bin/yt-dlp https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_linux
chmod +x ~/.local/bin/yt-dlp
```
</details>

## Use it as an MCP server

This is the part that matters. Wire it in once and any question about a video
becomes answerable.

**Claude Code**

```bash
claude mcp add framedeck -- python3 -m framedeck mcp
```

**opencode** — in `opencode.json`:

```json
{
  "mcp": {
    "framedeck": {
      "type": "local",
      "command": ["python3", "-m", "framedeck", "mcp"],
      "enabled": true
    }
  }
}
```

Then just ask:

> what's the layout of this invitation? https://vt.tiktok.com/ZSVKJNPcM/

The agent calls `framedeck_extract`, reads the returned frame paths as images,
and describes the eight sections it sees.

| Tool | Does |
|---|---|
| `framedeck_extract` | download + extract distinct frames, return the manifest |
| `framedeck_info` | caption, author, duration — no download |

The server speaks JSON-RPC 2.0 over stdio directly, without an MCP SDK. The
stdio framing is small enough that vendoring one would cost more than it saves,
and a dependency-free server drops into any sandbox unchanged.

## CLI

```
framedeck extract <url|path> [options]
  -o, --out DIR          output directory        (./framedeck-out)
  -n, --max-frames N     frame budget            (12)
  -w, --width PX         frame width             (720)
  -t, --threshold F      scene sensitivity, lower finds more  (0.12)
  -d, --distance N       dedupe strictness, lower keeps more  (10)
      --no-sheet         skip the contact sheet
      --keep-video       keep the downloaded video
      --json             print the manifest

framedeck info <url>     metadata only
framedeck mcp            run as an MCP server over stdio
```

Two knobs cover almost all tuning. Getting near-duplicates anyway? Raise
`-d`. Missing a screen that was briefly on camera? Lower `-t`.

## Manifest

```json
{
  "source": { "provider": "tikwm", "author": "editmateproject", "duration": 36.0,
              "title": "Cover undangan nggak harus foto prewed …" },
  "frame_count": 10,
  "contact_sheet": "/abs/out/contact-sheet.jpg",
  "frames": [
    { "index": 1, "time": 0.2, "path": "/abs/out/frame-01-t0000.20.jpg",
      "phash": "3c1e0f078381c0e0", "reason": "first" }
  ]
}
```

`reason` is `first`, `scene` (the picture changed here) or `anchor` (uniform
sample, kept because nothing else covered this stretch).

## Good for

- Reading a UI or design walkthrough posted as a video
- Pulling structure out of a demo reel a client sent instead of a spec
- Turning a screen recording of a bug into frames you can point at
- Any "what happens in this video" question aimed at an agent

## Limits

- Frames only — no audio, no transcript. Pipe the audio to whisper if you need
  words; framedeck deliberately stops at the picture.
- Downloads whatever the URL points at. Respect the platform's terms and other
  people's copyright: reference material is not the same as redistribution, and
  neither is it a licence to clone someone's work.

MIT.