9xvision

Video perception for AI agents. Give any agent (Claude Code, Codex, Cursor, anything that speaks MCP or can run a CLI) a video file or URL, and it gets back structured, timestamped evidence instead of a handful of random frames:

• Screen recordings (apps, websites): every UI state in order, distinct views, the transition between each (tap, scroll, sheet, swipe), OCR of every screen, stitched full pages, and a flow graph. Built for teardown work like "rebuild this onboarding".

• Everything else (films, music videos, talks, demos): shots split at camera cuts and content beats (slide changes inside one take), camera motion, palettes, colour script, on-screen text and speech per segment.

• Drill-down tools: zoom (dense frames of an interval), frame (full-res crop), diff (what changed between two moments), search (on-screen text and speech across everything watched).

Everything runs locally. There are no API keys and nothing is uploaded.

What you get

App and screen recordings: the flow, step by step, naming what changed (+ = turned on / selected, − = turned off):

00:09.3  V7 -> V8    [content change at +'Get recommendations' · +'Female']
00:10.4  V8 -> V9    [screen change]
00:17.7  V13 -> V14  [local change at −'Work' · −'Self-Care']
00:18.4  V14 -> V15  [local change at −'Hobbies']

Every distinct screen is kept full-res with its text, background and palette, and the list compresses repeats ("same as V9; added: 1986 | 1987 | 1988").

Everything else: one labelled cell per shot (or slide / scene change), with camera motion, palette, on-screen text and speech in INDEX.md, plus a colour script of the whole film:

Big Buck Bunny © Blender Foundation, CC BY 3.0.

Related MCP server: claude-real-video

Install

uv tool install git+https://github.com/ASGARD-STUDIOS/9xvision
9xvision setup

setup installs whatever is missing (ffmpeg, OCR, local speech-to-text; it asks first) and plugs 9xvision into every agent it finds: Claude Code, Codex, Cursor and Claude Desktop, as an MCP server plus the skill. Then restart the agent and ask it about a video. 9xvision doctor shows what's installed and connected; 9xvision setup --remove undoes it. Update later with uv tool upgrade ninexvision.

No uv yet? curl -LsSf https://astral.sh/uv/install.sh | sh (macOS/Linux). Manual routes: Claude Code /plugin marketplace add ASGARD-STUDIOS/9xvision then /plugin install 9xvision@9x; any MCP client: command 9xvision-mcp. Any other agent with a shell: put 9xvision on PATH and give it the text from 9xvision skill.

Benchmark

The same agent (Claude Code headless, Claude Sonnet 5.5) answered the same questions about the same videos with and without 9xvision. Both arms had the whole machine: ffmpeg, ffprobe, whisper.cpp, tesseract and python. The kit arm also had 9xvision and the skill text, and started from an empty cache each run, so its processing time counts. Answers were graded blind by Claude Opus 5.5 against answer keys written and verified from raw frames before the runs. 3–4 runs per cell, October 2026.

What this means, without spin:

• A frontier agent with ffmpeg is already a strong video reader. The kit's gains come from fewer misses on fast, visual-only detail (0.5 s inserts in a music video, a file path on a slide), fewer tokens and turns, and a reusable cache: every later question about the same video is instant.

• It produces things the baseline never builds: the UI flow with what changed at each step (local change at 'Work' · 'Self-Care'), every screen's copy, palettes, a colour script, and stitched pages.

• Cold start is the cost. A 74-minute video takes about 100 s to process the first time, so on a single question about a long, captioned video, caption search plus a few ffmpeg grabs is faster.

• Selection state: the benchmark found the agent sometimes misread checked vs unchecked options. Since then the flow marks each change +/− (gained or lost colour), which reads the DailyBean recording correctly; it is a heuristic, so diff confirms.

The answer keys were corrected three times after inspecting failures; each time the key was wrong, not the run, and both arms were re-graded. Harness and grader: bench/.

Use

9xvision watch recording.mp4            # prints INDEX.md, caches everything (~/.9xvision)
9xvision watch "https://youtu.be/..."   # URLs via yt-dlp
9xvision zoom <watch> 01:20 01:30 -n 16
9xvision frame <watch> 00:43.2 --crop 0,0.6,1,0.4
9xvision diff <watch> 00:17.0 00:18.3
9xvision search "limited offer"
9xvision look <watch> view:12

The agent reads INDEX.md, looks at the overview contact sheets, and drills down only where a question needs it. Every image is capped at 1568 px on the long edge, so it is never silently downscaled, and the image's token cost is printed.

How it works

One ffmpeg decode pass at 12 fps (6 fps over an hour) computes per-frame signals: HSV content change, changed-pixel fraction, phase-correlation shift, luma and sharpness. Status bars and home indicators are masked on phone recordings.

• Screen mode finds stable UI states with an adaptive quiet threshold and keeps screens that never hold still as "moving" states. It clusters revisits into views on sharper 480 px frames, so toggles and checkmarks count. Transitions are classified by motion and change area, scroll groups are merged, and pages are stitched.

• Film mode detects cuts with adaptive thresholds and fades. Content beats are added where the picture changes and holds without a cut. Keyframes are the sharpest frame in the middle of each shot.

• Text: Apple Vision OCR on macOS (in a watchdog process), tesseract elsewhere.

• Speech: captions first (yt-dlp or a sidecar .vtt), then whisper.cpp locally.

Limits

• Scroll stitching is experimental and only runs when consecutive states share content.

• Audio is transcribed, but music, sound design and tone are not described.

• OCR quality follows the source resolution; on 720p screen shares, small UI text needs frame --crop.

License

MIT © Koto Murai