shortform-analyzer

Break a short-form video into scenes, and say why each one keeps the viewer watching

Features • Benchmark • Get started • Usage • How it works • Building blocks

A real run of analyzeVideo on Zach King's “a message to all AI” (the video is his) · MP4

Give it a TikTok, Reel or Short and it tells you, scene by scene, what is said and shown, which beat the scene plays (hook, setup, value, proof, payoff, cta), what keeps the viewer watching and how strongly, and what format the video is. One line in Node, or a CLI, an MCP server and an agent skill.

import { analyzeVideo, summarize } from "shortform-analyzer";

const a = await analyzeVideo("clip.mp4");
if (a.ok) console.log(summarize(a));
// meme, awareness, 5 s, 3 scenes, 3 with text on screen. Made as text only, audio: music only.
//   hook 0.0-1.5 s (curiosity): "Wait"
//   value 1.5-3.0 s (demonstration) (nothing said)
//   cta 3.0-4.5 s (demonstration): "Follow!"

Features

• Scene by scene: cuts, the words said in each scene, its middle frame, and a beat, lever and 0-4 hold score per scene.

• Fast and light: one transcription and one decision request per video. About 3 to 9 seconds and under 70 MB of memory (benchmark).

• Never throws: a bad video or a missing key comes back as { ok: false, error }. A refused answer falls back to a stated default.

• Your own questions: up to 20 yes/no questions per video, or any question per scene through the building blocks.

• Four ways in: a library, a CLI, an MCP server for Claude, Cursor and others, and an agent skill.

• Readable output: analysis.json with a JSON Schema, a one-page report.html, and sheet.jpg with every frame on one image.

Related MCP server: clipspeed-mcp

Benchmark

Three public TikToks through four tools, all on the same model (gpt-6-luna), on an Apple Silicon laptop, October 2026. Median of 2 runs (video-analyzer: 1 run).

Why it is faster: one Decisions request labels every scene in 0.7-1.2 s, where the same model writing its answer as chat takes 7-18 s, and one frame per scene sends 2-4 times fewer images. claude-real-video costs a little less because it transcribes on your machine, paying with 9-44 s of CPU and about 1 GB of memory instead. Most of our time and cost is the whisper-1 call.

• Videos: @pet..supplies' cat brush (17 s, product demo), Zach King, "a message to all AI" (28 s, many cuts), GaryVee (61 s, talking head). They belong to their creators and are not in this repo.

• claude-video-vision (1.4.0) and claude-real-video (ab4b9b2) stop at frames and a transcript, so each gets one Chat Completions request asking for the same breakdown. video-analyzer (2b095fa) needs PyAV below 19 for its Whisper to run; gpt-6-luna rejects its temperature and max_tokens, which were dropped and renamed.

• Prices: gpt-6-luna $0.10 per million input tokens and $0.50 per million output tokens, Decisions output free, whisper-1 $0.006 a minute. Install size is node_modules after npm install.

• Speed, cost and resources only. The quality of the labels is not compared.

To run it yourself, on stand-in videos or your own: bench/. The numbers above are in bench/results/2026-10.json.

Get started

You need:

• Node.js 20 or later;

• ffmpeg and ffprobe on PATH (brew install ffmpeg, apt install ffmpeg), or pass ffmpegPath and ffprobePath;

• an OpenAI key in OPENAI_API_KEY.

npm install shortform-analyzer

Or run it without installing:

npx shortform-analyzer analyze clip.mp4

Usage

Library

import { writeFile } from "node:fs/promises";
import { analyzeVideo, reportHtml, summarize } from "shortform-analyzer";

const a = await analyzeVideo("clip.mp4", {
  questions: [{ name: "face_on_camera", question: "Is a person's face on camera at any point?" }],
});
if (!a.ok) throw new Error(a.error);

a.scenes[0];          // { index, start, end, thumbnail, transcript, beat, beatProbs, lever, leverProbs, hold, textOnScreen }
a.sections;           // consecutive scenes with the same beat
a.format;             // "review", "tutorial", "meme", …
a.answers;            // { face_on_camera: 0.04 }
console.log(summarize(a));                       // text for an agent
await writeFile("report.html", reportHtml(a));   // one self-contained page

Options: openaiApiKey (default OPENAI_API_KEY), openaiBaseUrl (default OPENAI_BASE_URL, else https://api.openai.com/v1; Azure OpenAI works through it), questions (up to 20), ffmpegPath, ffprobePath, fetch.

The output's fields are in docs/reference.md and schema/analysis.schema.json (schemaVersion: 1).

CLI

npx shortform-analyzer analyze clip.mp4
npx shortform-analyzer analyze clip.mp4 --question face="Is a person's face on camera?" --json
npx shortform-analyzer screen clip.mp4 --question face="Is a person's face on camera?"

analyze prints the summary (or the JSON with --json) and writes <file>.analysis/:

• analysis.json and report.html;

• frames/01.jpg…: the 512 px frames the model saw;

• sheet.jpg: every frame tiled 4 wide, so an agent can look at one image.

Exit codes: 0 ok, 1 the analysis failed (the reason is on stderr), 2 a usage error.

MCP server

Claude Code:

claude mcp add shortform-analyzer -e OPENAI_API_KEY=sk-... -- npx -y shortform-analyzer mcp

Claude Desktop, Cursor and other clients:

{
  "mcpServers": {
    "shortform-analyzer": {
      "command": "npx",
      "args": ["-y", "shortform-analyzer", "mcp"],
      "env": { "OPENAI_API_KEY": "sk-..." }
    }
  }
}

Agent skill

skills/shortform-analyzer/SKILL.md teaches an agent when to run it and how to read the result. It ships in the npm package too.

npx skills add trevkang/shortform-analyzer

How it works

One video goes through four steps (the animation above shows steps 1 and 4):

                       ┌─ extractFrames ─────────────────────┐
video ─ probe ─ detectScenes                                 ├─ sceneEvidence ─ decide ─ parseShortform ─ Analysis
                       └─ extractAudio ─ transcribe ─ alignWords ┘                 ▲
                                                                     shortformQuestions (or yours)

The words are given to the scenes before the decision, because the model reads them side by side. And the decision is one request for the whole video, not one per scene, because a scene's role depends on its neighbours: the same shot is a hook at 0 s and a payoff at 20 s.

The short-form labels are just one set of questions. Steps 1 to 4 know nothing about short-form video, so you can use them on their own or ask your own questions per scene. The layers are explained in docs/concepts.md.

Building blocks

Every step is public from shortform-analyzer/primitives.

Scenes and the words said in each, with no model call:

import { probe, detectScenes, extractAudio, transcribe, alignWords } from "shortform-analyzer/primitives";

const { duration } = await probe("talk.mp4");
const spans = await detectScenes("talk.mp4", { duration, maxTake: 30 });
const { words } = await transcribe(await extractAudio("talk.mp4"));
const texts = alignWords(words, spans);

The short-form preset plus a question of your own, asked of every scene:

import { decide, extractFrames, parseShortform, sceneEvidence, shortformQuestions, type DecisionQuestion } from "shortform-analyzer/primitives";

const frames = await extractFrames("clip.mp4", spans);
const questions: DecisionQuestion[] = [
  ...shortformQuestions(),
  { type: "predicate", name: "product", scope: "scene", instructions: "Is the product visible in scene {scene}?" },
];
const d = await decide(sceneEvidence(spans, frames, texts, { duration }), questions);
const labels = parseShortform(d);                                    // beats, levers, hold… with fallbacks
const product = d.scenes.map((s) => s.product?.probability ?? 0.5); // your answer, per scene

decide takes predicate (a probability), choice (one of your options, with probabilities) and score (0 to the number of levels − 1) questions, for the whole video or per scene, up to 400 per request. It returns only what the model answered: a refused answer is absent, and you choose the fallback. The building blocks throw an Error that names the step (video:, transcribe:, decide:); the functions above them never throw.

The labels

• Levers: curiosity, pattern_interrupt, relatable, demonstration, social_proof, emotion, direct_ask, none.

• Hold: 0 (most viewers scroll away) to 4 (nearly impossible to scroll away).

• Formats: meme, skit, product_demo, tutorial, review, storytime, before_after, unboxing, behind_the_scenes, talking_head, other. goal is conversion when sells ≥ 0.5, else awareness.

• Production: on_camera_person, screen_recording, hands_only, broll_voiceover, text_only, animation, mixed.

• Voice: on_camera_speech, voiceover, none, music_only.

When the model refuses or skips an answer: beat value, lever none, hold 2, format other, production mixed, voice none, a question 0.5. A video without speech is still analysed from its frames, with an empty transcript and transcriptError. Definitions and thresholds are in docs/video-analysis.md.

Keys and costs

Each video takes one whisper-1 transcription and one Decisions request (gpt-6-luna) on your OPENAI_API_KEY: from $0.0025 for a 17 s video to $0.0078 for a 61 s one (benchmark). There is no telemetry, and your key goes only to OpenAI.

Development

pnpm install
pnpm test           # builds, then runs every test (videos are made with ffmpeg lavfi; no network)
pnpm typecheck
pnpm test:consumer  # installs the packed tarball into a blank project and imports it

Start with docs/concepts.md; a change starts as a doc edit, then a test, then the code.

Credits

Built by Trevor and Viktor. It started as the video analysis module of viral-oracle, their team project at the Omnara hackathon (October 2026).

The how-it-works animation is rendered with render.mjs from Barty-Bart/motion-graphics (MIT); the video in it belongs to Zach King; Geist fonts are under the SIL Open Font License (docs/media).