vidx
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@vidxindex ~/Movies/talk.mp4, cut the dead air, and make a 9:16 reel where I talk about pricing"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
vidx
Let AI agents edit video without burning tokens on frames.
For an agent to "watch" a 10-minute video at 1 frame per second costs ~67,000 tokens; an hour is ~400,000.
vidx watches it once, locally, and gives the agent a compact text map instead: shots, transcript,
silences, laughs, loud moments and text-searchable visuals. The agent reads a ~600-token overview,
searches what it needs, edits with a few lines of script, renders with ffmpeg and runs automatic QA.
Pixels reach the agent only when it explicitly asks to look (peek).
Runs 100% locally. No paid APIs. Works with any MCP agent (Claude Code, Claude Desktop, …).
Measured on 3 real videos (8, 10, 60 min) | transcript + frames | vidx |
Get the gist of the video | 7k–17k tokens | ~650–930 |
Find a moment, read it, look at it | 7k–17k | ~1.4k–1.8k |
Cut dead air, render, QA | 3k–8k | ~0.8k–1k |
Overview of a 10-min vs 60-min video | grows with length | 507 vs 564 |
6–18× fewer tokens in these tests, and the gap widens with longer video. Details, method and the cases where it does not win: TOKEN_REPORT.md.
Quickstart (Claude Code)
brew install ffmpeg uv # Linux: apt install ffmpeg, then curl -LsSf https://astral.sh/uv/install.sh | sh
claude mcp add vidx -- uvx --from git+https://github.com/renjithrrkj/vidx vidx-mcpNo clone needed. Then ask your agent something like "index ~/Movies/talk.mp4, cut the dead air and make a 9:16 reel of the part where I talk about pricing".
The first index downloads Whisper small (~460 MB) and CLIP ViT-B-32 (~600 MB). Indexing takes
CPU time, not tokens: about 10 min per hour of video on an M1, under ~1 GB of RAM.
Claude Desktop / other MCP clients: add to your MCP config (desktop apps don't inherit your shell PATH, so give full paths):
"mcpServers": {
"vidx": {
"command": "/full/path/to/uvx",
"args": ["--from", "git+https://github.com/renjithrrkj/vidx", "vidx-mcp"],
"env": { "PATH": "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" }
}
}From source (to hack on it):
git clone https://github.com/renjithrrkj/vidx && cd vidx && uv sync
claude mcp add vidx -- uv run --directory "$PWD" vidx-mcpTested on macOS (Apple silicon). Linux should work (CPU/CUDA); Windows is untested.
Related MCP server: klaket-mcp
The tools
Tool | What it returns |
| video_id, duration, shots, language, short summary |
| table of contents (scenes cover the whole video in ≤limit lines) |
| ranked time ranges: speech / visual (CLIP) / event ( |
| words, events, shots for one range ( |
| image: one contact sheet, ≤4×4 labeled tiles |
| version id + segment count, output duration, warnings |
| output path, duration, size |
| QA report ≤15 lines, |
| added / removed / moved source ranges, duration delta |
Edit DSL
base all # start from the full video (default) | base none
drop 30.0-31.2 # remove a time range (also 1:02.5 style)
drop s7 # remove a shot (or s7-s9)
drop silence>1.2 # remove every silence longer than 1.2s (also: black, freeze, loud)
keep 120.0-150.0 # add a range (with base none; line order = output order)
keep s3
snap words # move cut points out of words into the nearest gap
pad 0.15 # handles around kept segments (never into a neighbouring word)
trans xfade 0.3 # crossfade at every join (default: hard cut)
crop 9:16 center
speed 1.1Typical dead-air cleanup: drop silence>1.0 / snap words / pad 0.1.
edit(..., parent_version="v3") appends to v3's script, so fixes from check are one-liners.
Presets: original (as is), reel_9x16 (1080×1920, −16 LUFS, ≤90 s), yt_16x9 (1920×1080, −14 LUFS).
CLI (same output as the tools)
uv run vidx index ~/Movies/talk.mp4
uv run vidx overview 5450c7a6
uv run vidx query 5450c7a6 "whiteboard diagram" --kind visual
uv run vidx segment 5450c7a6 120 150
uv run vidx peek 5450c7a6 "12.5,s3,40" --out sheet.jpg
uv run vidx edit 5450c7a6 "drop silence>1.0;snap words;pad 0.1"
uv run vidx render v1 --preset reel_9x16
uv run vidx check v1
uv run vidx diff v1 v2
uv run vidx usage # characters returned per call (~tokens)Optional extras
uv sync --extra captions→index_video(..., captions=true)adds Florence-2 one-line shot captions.uv sync --extra otio→ every render also writes a.otiotimeline (opens in Resolve / Premiere).laughevents (heuristic audience-reaction detector) run by default:query(id, "laugh").
Measured (Apple M1, 8 GB, CPU whisper + MPS CLIP)
Video | Index time | Overview size |
10:00 1080p vertical stand-up set (1 take, Malayalam) | shots 71 s, visual 23 s, transcript 120 s, audio 64 s ≈ 4.6 min | 2.6k chars (~650 tok) |
7:51 720p short film (124 shots, letterboxed, Malayalam) | 2.6 min (+5 min with captions) | 3.0k chars (~760 tok) |
A whole session on the 10-min video (23 calls: overviews, queries, segments, edits, 2 peeks) returned ~9.4k chars, roughly 2.3k tokens of text. Rendering the 9:41 silence-cut at 1080×1920 takes ~3 min, a 54 s reel ~10 s.
Malayalam and other languages that Whisper-small transcribes badly are indexed as an English
translation (flagged in the summary). Set VIDX_WHISPER_MODEL=large-v3 for native-script transcripts.
Config (env vars)
VIDX_HOME (default ~/.vidx), VIDX_WHISPER_MODEL (small; try medium for Malayalam),
VIDX_WHISPER_LANGUAGE (force e.g. ml), VIDX_WHISPER_TASK (auto/transcribe/translate), VIDX_DEVICE (cuda/mps/cpu),
VIDX_ENCODER (libx264 or h264_videotoolbox), VIDX_SHOT_THRESHOLD, VIDX_SILENCE_DB.
Layout
src/vidx/
analyzers/ probe, shots, visual (CLIP + tags + windows), transcript, audio events (+letterbox),
laugh, captions (optional) — pluggable list
store.py SQLite (~/.vidx/vidx.db) + per-video .npy embeddings
query.py overview / query / get_segment / peek
dsl.py edit script -> timeline (pure segment math)
render.py timeline -> ffmpeg
checks.py QA diff.py version compare
api.py the 9 operations (+ usage log) cli.py / mcp_server.py thin wrappers
otio_export.py .otio next to rendersAdding an analyzer: write a class with name, optional, and run(video, ctx) -> AnalyzerResult
that writes rows to shots/words/events (or files in ~/.vidx/<id>/), then append it in
analyzers/__init__.py:registry().
Tests
uv run pytest -m "not slow" # DSL, segment math, snap/pad, diff, checks (no video needed)
uv run pytest # + end-to-end on a generated testsrc clipSee DECISIONS.md for the choices made along the way.
License
MIT, see LICENSE.
Available Tools
9 toolscheckA
QA a rendered version: black frames, new freezes, mid-word cuts, audio jumps at joins, loudness vs preset target, duration limits. Each finding has a time and a DSL fix. Returns OK when clean.
| Name | Required | Description | Default |
|---|---|---|---|
| version_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and correctly conveys that output is findings with a time and a DSL fix, plus an OK signal when clean. However it never states whether the check is read-only/idempotent, whether it can be slow or costly (analysis of rendered media), or whether the DSL fix is applied or merely suggested.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded verb+object, then a compact colon list of check categories, closing with the success condition. No filler sentences and nothing redundant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-value explanation is not required, and the description still notes the finding shape and the OK result. The remaining gap is operational context — prerequisites, read-only nature, and cost — which is modest for a single-parameter QA tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is one parameter (version_id) and schema coverage is 0%, so the description must compensate. 'A rendered version' loosely ties the argument to a rendered version object, but gives no format hint (ID vs name) beyond what the self-explanatory parameter name already implies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific action (QA a rendered version) and enumerates the exact defect classes it looks for, so an agent can tell it apart from 'render' or 'diff'. It does not name siblings, but the scope is unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied by 'a rendered version', which suggests post-render inspection, and by 'Returns OK when clean', which hints at a pass/fail gate. No explicit when-to-use, no prerequisites (must a render exist first?), and no alternatives such as 'diff' for version comparison.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
diffC
What changed between two versions: added/removed/moved source ranges, option changes, output duration delta.
| Name | Required | Description | Default |
|---|---|---|---|
| version_a | Yes | ||
| version_b | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, yet it discloses no behavioral traits such as read-only safety, permission requirements, failure modes, or cost. What it does describe (the diff categories) is return content, which is already covered by the output schema, so it adds little behavioral value.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single compact sentence, front-loaded with the operation and followed by the specific diff dimensions. It is appropriately sized, though the clipped colon-list style is slightly terse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complexity is low (two flat string params, output schema present so returns need not be explained), but the total absence of parameter meaning and of any usage routing leaves a real gap for an agent deciding when and how to call it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% and the two required parameters (version_a, version_b) are documented nowhere. The description never explains what form a 'version' identifier takes (ID, label, timestamp) or whether ordering matters, leaving the agent unable to populate the call confidently.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific comparison operation ('what changed between two versions') and enumerates the diff categories it produces (source ranges, option changes, duration delta), so the agent can tell it is a comparison/diff tool rather than an editor or renderer. It does not name or distinguish any sibling explicitly, keeping it at 4 rather than 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to use this tool versus siblings like overview, query, or get_segment, nor any prerequisite or exclusion. The agent must infer from the name alone that this is the version-comparison entry point.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
editA
Create a version from an edit script (see server instructions for the DSL). With parent_version,
the script is appended to the parent's script (incremental fix) unless it has its own base line.
Returns version id, segment count, output duration and warnings.
| Name | Required | Description | Default |
|---|---|---|---|
| dsl | Yes | ||
| video_id | Yes | ||
| parent_version | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well: it explains the incremental-append behavior when parent_version is set, the escape hatch when the script declares its own base line, and what is returned (version id, segment count, duration, warnings). It omits error behavior and whether existing versions are affected, which is why it is not a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with no filler, and the core action is front-loaded before the parent_version nuance and return summary. The parenthetical pointer to server instructions is brief and earns its place, though it does offload essential context elsewhere.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so the description need not enumerate return fields in depth, and it still names the main ones. Combined with the DSL pointer and append semantics, an agent has enough to call this correctly, though the dependency on external server instructions is a small completeness gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It adds real meaning for parent_version (appended to the parent's script unless a base line exists), but video_id and dsl are left entirely to the schema, with dsl deferred to external server instructions rather than described here.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: it creates a version from an edit script, which distinguishes it from siblings like render, check, and diff. However, the bare name 'edit' plus the fact that the distinguishing work is done by the DSL reference rather than explicit sibling differentiation keeps this just below a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an implied usage rule for parent_version (append vs. base-line override) and points to server instructions for the DSL, which is useful. But it never states when to use this tool versus render/check/diff, nor any prerequisite ordering (e.g., whether a render or check is required afterward).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_segmentA
Detail for one range: start/end in seconds or shot ids (start="s3" alone = that shot). fields: any of words,events,shots,wordtimes (wordtimes = per-word timestamps for precise cuts).
| Name | Required | Description | Default |
|---|---|---|---|
| end | No | ||
| start | Yes | ||
| fields | No | words,events,shots | |
| video_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It does disclose input addressing formats and what 'wordtimes' yields, and read-only behavior is implied by 'Detail', but it says nothing about output size, cost, or any constraints on ranges.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two compact sentences, front-loaded with the core operation and then parameter help; every clause carries information, with no preamble or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be described, and the description fills the format gaps the bare schema leaves. Remaining gaps are minor (video_id semantics, what an omitted end defaults to), so it is nearly complete for a 4-parameter read tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it largely does: start/end are explained as seconds or shot ids with a concrete example ('start="s3"'), and valid fields values are enumerated with wordtimes' meaning given. Only video_id and the default end behavior remain unexplained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a concrete operation ('Detail for one range') on a named resource (a video range/segment), with the start/end addressing scheme spelled out. It does not explicitly contrast itself with siblings like peek or query, so an agent must infer the boundary, but the purpose itself is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: 'Detail for one range' suggests fetching full detail for exactly one span, and 'for precise cuts' hints at the wordtimes use case. No when-to-use/when-not or alternative tool is named, so the agent still has to guess versus peek/query/overview.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
index_videoA
Index a video file (shots, transcript, audio/visual events, CLIP embeddings). Re-indexing the same file is a no-op unless force=true. captions=true adds one-line shot captions (slower). Returns video_id, duration, shot count, language and a short summary.
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | ||
| force | No | ||
| captions | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden and does real work: it discloses idempotency semantics (no-op on repeat, force overrides), a cost trade-off (captions=true is slower), and the shape of the result. It does not mention permission requirements, whether the call is long-running/blocking, or failure modes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences with zero filler, and the most decision-relevant facts (what it indexes, then the no-op/force rule) are front-loaded before the secondary captions caveat and the return summary.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutation-style tool with no annotations, the description covers the essentials an agent needs: what gets created, when nothing happens, and the cost of an option. Since an output schema exists, restating return fields is optional; the remaining gap is operational context such as expected runtime or permissions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it does for the two non-obvious flags: force's override-of-idempotency behavior and captions' effect plus slowdown. The required 'path' parameter is left with no format guidance (accepted file types, absolute vs relative), so coverage is incomplete rather than absent.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Index) and resource (video file) and enumerates the artifacts produced: shots, transcript, audio/visual events, CLIP embeddings. That scope is specific enough to separate it from siblings like overview, query, or peek, which operate on already-indexed videos.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clearly establishes the context of use: indexing is the step that populates the index, and re-running it is a no-op unless force=true, which tells the agent when to set that flag. It stops short of naming sibling alternatives or stating prerequisites (e.g., that query/get_segment depend on this having run).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
overviewA
Table of contents. level="scenes" covers the whole video in <=limit lines (similar shots merged, long shots split at pauses); level="shots" lists shots s1..sN (page with offset).
| Name | Required | Description | Default |
|---|---|---|---|
| level | No | scenes | |
| limit | No | ||
| offset | No | ||
| video_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations, so the description carries the burden. It does disclose non-obvious behavior: shot merging/splitting at pauses for the scenes level and offset-based paging for the shots level. It never states this is a read-only operation or what the response contains, leaving a gap for an annotation-free tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense sentences with the head concept front-loaded and no filler. The parenthetical clauses are information-bearing rather than padding, and it stays short despite covering two modes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be described, and the level/limit/offset behavior is covered. What is missing is any positioning against the eight sibling tools and any read-only confirmation for a tool with zero annotations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it does: it defines the two meaningful values of level and explains that limit caps the scenes output while offset pages the shots output. Only video_id and the numeric defaults remain unexplained, but those are self-evident.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a concrete metaphor ("Table of contents") immediately qualified by what each level returns: scenes merge similar shots and split long ones, shots enumerate s1..sN. An agent understands this lists the video's structural units. It does not, however, differentiate itself from siblings like index_video or peek.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied through the level semantics (scenes = whole-video summary, shots = full enumeration). There is no explicit statement of when to pick this over index_video, peek, or query, and no exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
peekA
SEE frames (costs image tokens; use sparingly). target = video_id or a rendered version id (v3). times = seconds or shot ids, e.g. ["12.5", "s3"]; omit for an even visual summary. Returns one contact sheet (max 4x4 tiles, each labeled with its timestamp); size = longest side of a tile.
| Name | Required | Description | Default |
|---|---|---|---|
| grid | No | ||
| size | No | ||
| times | No | ||
| target | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does meaningful work: it discloses the cost model (image tokens), the return shape (one contact sheet, max 4x4 tiles, each labeled with its timestamp), and defaults via the times parameter. It stops short of stating safety/read-only semantics or what happens when the target is invalid.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense and front-loaded, leading with the action and the cost caveat before parameter semantics. Every sentence carries information, though the parameter glosses run together in a single unbroken block rather than being cleanly separated.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-parameter tool with no annotations and no output schema, the description covers the target, times, size, the cost model, and the return format, which is most of what an agent needs. The unexplained grid parameter and lack of stated failure behavior keep it from being wholly complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it largely does: target is explained as video_id or a rendered version id, times as seconds or shot ids with omitted-meaning explanation, and size as the longest side of a tile. Only grid is left entirely unexplained, which is a notable gap for a parameter whose default is true.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (SEE) and resource (frames), and immediately clarifies the output is a single labeled contact sheet, which separates it from text-oriented siblings like query or get_segment. It does not explicitly name or contrast against sibling tools such as overview or get_segment, so the differentiation is implied rather than stated.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"costs image tokens; use sparingly" gives a clear caution about when to be cautious with this tool, which is real guidance. However, it names no alternative among the eight siblings (e.g., when to prefer get_segment or overview over peeking), so the agent must infer the selection criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
queryB
Find time ranges. kind: speech (exact+fuzzy transcript match), visual (CLIP text->shot), event (e.g. "silence>1.5", "black", "loud", "freeze"), any (event syntax, else speech+visual).
| Name | Required | Description | Default |
|---|---|---|---|
| kind | No | any | |
| text | Yes | ||
| limit | No | ||
| video_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It does disclose matching behavior ('exact+fuzzy transcript match', 'CLIP text->shot', 'silence>1.5', 'freeze'), which is genuine behavioral context, but it omits that this is a read-only search, how many results come back (limit is never mentioned), and any pagination or ordering behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Dense and front-loaded: the purpose leads, then each kind is defined inline with no filler sentences. It borders on cryptic for the opening clause ('Find time ranges'), which costs a point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need no explanation, and the four kinds are well covered. However, limit/pagination behavior is absent, and for a tool whose output feeds siblings like get_segment, the description leaves the agent without enough to call it confidently beyond the default kind='any'.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does explain the semantically complex parameter (kind) in detail, including accepted values and fallback behavior, plus event syntax examples — but text, video_id and especially limit (default 10, undocumented) receive no explanation, so the compensation is only partial.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The verb+resource pair ('Find time ranges') is terse and never states that the search operates over the video identified by video_id, though the kind definitions (transcript match, CLIP text->shot) let an agent infer it is a content search. It does not differentiate itself from siblings like get_segment or overview, so the agent must guess where this fits in the workflow.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied through the kind semantics: 'any' falls back to event syntax then speech+visual, and the event examples hint at when event matching applies. There is no explicit when-to-use/when-not guidance and no routing toward siblings such as get_segment for retrieving the actual matched range.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
renderC
Render a version with ffmpeg. preset: original | reel_9x16 | yt_16x9. Returns path, duration, size.
| Name | Required | Description | Default |
|---|---|---|---|
| preset | No | original | |
| out_path | No | ||
| version_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It hints that ffmpeg is invoked and that a file is produced, but says nothing about whether rendering is blocking or long-running, whether it overwrites out_path, or what permissions are required — meaningful gaps for a media-encoding mutation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the action, then the enum values, then the return shape — no filler. It is terse to the point that 'version' remains undefined jargon, which slightly undercuts the economy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists so return values needn't be spelled out (and the description repeats them anyway). The real gaps are behavioral: no indication of async vs blocking execution, overwrite behavior, or the meaning of the required version_id, which an agent needs before invoking a render job.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% and the schema has no enum for preset, so listing 'original | reel_9x16 | yt_16x9' is genuinely additive information the agent cannot get elsewhere. However, version_id and out_path are left completely unexplained, including what an empty default out_path means.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Render') plus resource ('a version') and names the underlying engine (ffmpeg), so the agent knows this produces a media file rather than metadata. It does not differentiate itself from siblings like edit or check, but the operation is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to call render versus the sibling tools (edit, check, peek, diff), nor any prerequisite such as needing an existing version_id. The agent must infer usage entirely from the name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
9 tool updates
v0.1.0- First observed
check - First observed
diff - First observed
edit - First observed
get_segment - First observed
index_video - First observed
overview - First observed
peek - First observed
query - First observed
render
TDQS
Scored across 9 tools
Each tool maps to a distinct stage of the video workflow: ingest (index_video), navigation (overview), search (query), detail (get_segment), visual inspection (peek), editing (edit), output (render), QA (check), and comparison (diff). The retrieval tools (overview/query/get_segment) could superficially overlap, but the descriptions clearly separate navigation, time-range search, and single-range detail.
All names are lowercase snake_case, which is stylistically consistent, and the two multiword names (index_video, get_segment) follow verb_noun. However, the rest are bare nouns or verbs (overview, query, peek, edit, render, check, diff), so the verb_noun pattern isn't uniform across the set.
Nine tools is well-scoped for a video indexing/editing/rendering pipeline, with each tool earning its place across a clear ingest-to-QA lifecycle. Nothing feels redundant or padded.
The surface covers the full lifecycle: indexing, TOC/navigation, search, detail retrieval, visual inspection, scripted editing, rendering with presets, QA checks, and version diffing. No obvious dead ends given the described domain.
Maintenance
Related MCP Connectors
Give AI random access to video: timestamped contact sheets + zoom into any start/end range.
Orccut: a real timeline video editor for AI agents - journaled edits, FFmpeg/MLT rendering
FFmpeg as a service for AI agents: typed video editing tools, async jobs, downloadable outputs.
Edit video by talking to your AI — search footage, cut timelines, apply effects, add captions.
Related MCP Servers
- AlicenseAqualityCmaintenanceConverts video into timestamped contact sheets, enabling AI agents to navigate and retrieve visual evidence from specific timecodes rather than watching entire clips.144 npm2Apache 2.0
- AlicenseAqualityDmaintenanceLet AI agents watch videos: local transcripts, speaker labels, scenes, chapters and exact-moment search from any video URL or file. Fully local, no API keys.43AGPL 3.0
- AlicenseNot gradedqualityBmaintenanceEnables LLM agents to process local videos into timestamped, citable text documents and then query them through tools for listing videos, retrieving transcripts, and fetching specific segments, all fully offline.MIT
- AlicenseAqualityBmaintenanceEnables AI agents to analyze videos by producing timestamped transcripts, keyframe contact sheets, and metadata reports from local files or online URLs, fully offline and without API keys.6MIT