Skip to main content
Glama
kwacky1

video-understanding-mcp

by kwacky1

video-understanding-mcp

Local stdio MCP server for extracting bounded, reproducible evidence from video.

Early development: Milestones 1 and 2 provide safe video inspection and offline transcription. Timestamped frame extraction is planned next.

The server provides:

  • video_probe, backed by ffprobe

  • video_transcribe, backed by FFmpeg and whisper.cpp

  • timestamped transcript JSON and Markdown written to caller-selected roots

  • absolute-path and allowed-root enforcement

  • symlink-escape protection

  • bounded, cancellable subprocess execution

  • content-addressed probe and transcript caching

  • vu-doctor readiness checks for FFmpeg, FFprobe, and whisper.cpp

  • synthetic media fixtures and MCP stdio contract tests

Requirements

  • Node.js 22 or newer

  • ffmpeg

  • ffprobe

  • whisper-cli

  • a local whisper.cpp GGML model

Install whisper.cpp and bootstrap a model

On macOS with Homebrew:

brew install whisper-cpp
npm run bootstrap:model
export VU_WHISPER_MODEL_PATH="$HOME/.local/share/video-understanding-mcp/models/ggml-large-v3-turbo-q5_0.bin"

The bootstrap script downloads the official ggml-large-v3-turbo-q5_0.bin model and verifies its Git LFS SHA-256 digest, 394221709cd5ad1f40c46e6031ca61bce88931e6e088c188294c6d5a55ffa7e2, before reporting the configuration value. A failed checksum leaves no accepted model. Pass a different destination directory when required:

npm run bootstrap:model -- /absolute/model/directory

The server never downloads a model or makes any other network request while processing media.

Related MCP server: klaket-mcp

Build and run

git clone https://github.com/kwacky1/video-understanding-mcp.git
cd video-understanding-mcp
npm install
npm run build
VU_ALLOWED_READ_ROOTS="$HOME/Videos" \
VU_ALLOWED_WRITE_ROOTS="$HOME/Documents/transcripts" \
VU_WHISPER_MODEL_PATH="$HOME/.local/share/video-understanding-mcp/models/ggml-large-v3-turbo-q5_0.bin" \
npm start

VU_ALLOWED_READ_ROOTS is a path-delimited list. If it is omitted, the server allows reads only within its current working directory.

Optional configuration:

Variable

Default

Purpose

VU_ALLOWED_READ_ROOTS

current directory

Readable filesystem roots

VU_ALLOWED_WRITE_ROOTS

readable roots

Writable transcript roots

VU_MAX_INPUT_BYTES

10737418240

Maximum accepted input size

VU_FFPROBE_PATH

ffprobe

FFprobe executable

VU_FFMPEG_PATH

ffmpeg

FFmpeg executable

VU_WHISPER_PATH

whisper-cli

whisper.cpp executable

VU_WHISPER_MODEL_PATH

none

Absolute path to a local GGML model

VU_CACHE_DIR

platform cache directory

Content-addressed cache root

Both root variables are path-delimited lists (: on macOS/Linux, ; on Windows). Output directories must already exist, resolve inside a configured writable root, and be writable.

Transcription

video_transcribe accepts:

{
  "path": "/absolute/path/demo.mp4",
  "output_dir": "/absolute/path/transcripts",
  "language": "en"
}

It extracts the first audio stream to a temporary 16 kHz mono WAV, runs whisper-cli locally, and removes scratch files on success, failure, or cancellation. The result includes timestamped segments and durable transcript_json_path and transcript_markdown_path values.

The transcript cache key includes the input SHA-256, transcription schema, canonical parameters, whisper executable fingerprint, and model SHA-256. Repeating the same call reuses cached transcript artefacts and returns cache_hit: true; changing the input, language, model, or executable configuration invalidates the cache.

Doctor

npm run doctor
npm run doctor -- --json

The doctor exits non-zero until all three executables are ready. Whisper is reported now so Milestone 2 prerequisites are visible before transcription is added.

Licence

MIT

Available Tools

2 tools
video_probeProbe local videoA

Inspect a local media file with ffprobe and return normalised metadata. The path must be absolute and inside an allowed read root.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYesAbsolute path to a local media file

Output Schema

ParametersJSON Schema
NameRequiredDescription
streamsYes
cache_hitYes
containerYes
duration_msYes
input_sha256Yes
schema_versionYes
ffprobe_versionYes

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must stand alone. It does convey a read-only inspection via ffprobe, but it does not explicitly state that the file is unmodified, require specific permissions, or warn about failure modes. It is minimally transparent but not rich.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, each earning its place: the first defines the operation, the second clarifies the precondition. The wording is direct, efficiently front-loaded, and contains no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has an output schema, so the return value is already documented. With only one parameter and clear purpose, the description covers the key constraint (path root). It does not address branch conditions for failure or permission, but it is sufficiently complete for a simple read-only inspection tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema describes 'path' as an absolute path to a local media file, but the description adds an important constraint: the path must be inside an allowed read root. This goes beyond the schema and adds meaningful semantic guidance for correct invocation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific action (inspect), resource (local media file), method (ffprobe), and output (normalised metadata). This plainly differentiates it from the sibling video_transcribe, which would perform transcription rather than metadata inspection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is given about when to choose this tool over video_transcribe. The description mentions an execution constraint (absolute path, allowed read root) but does not address selection among alternatives or any prerequisite/context beyond that.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

video_transcribeTranscribe local videoA

Transcribe the first audio stream of a local media file with whisper.cpp. Processing is offline and writes timestamped JSON and Markdown into an allowed output directory.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYesAbsolute path to a local media file
languageNoWhisper language code, or auto for detectionen
output_dirYesExisting absolute directory for durable transcript files; the server does not create it

Output Schema

ParametersJSON Schema
NameRequiredDescription
engineYes
languageYes
segmentsYes
cache_hitYes
duration_msYes
input_sha256Yes
schema_versionYes
transcript_json_pathYes
transcript_markdown_pathYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the behavioral burden. It discloses that processing is offline and that the tool writes timestamped JSON and Markdown files, which conveys privacy implications and side effects. It does not detail overwrite behavior or error cases, but the key operational behavior is covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences with no filler. The primary action and constraints are front-loaded, and every clause adds meaningful information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description captures the essential context: local file input, whisper.cpp engine, offline processing, and durable outputs. An output schema exists, so return-value documentation is not required here. Minor ambiguity around 'allowed output directory' is acceptable since the schema already notes the server does not create it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all three parameters well. The description adds no parameter-level detail, but it doesn't need to because the input schema covers path, output_dir, and language semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Transcribe the first audio stream of a local media file with whisper.cpp.' This is clearly distinct from the sibling video_probe, which implies probing metadata rather than producing transcripts.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear context: use this tool when you need an offline transcription of a local media file written to timestamped output files. It does not explicitly say 'use video_probe instead for metadata' or list exclusions, so it stops short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv0.1.0
    • First observedvideo_probe
    • First observedvideo_transcribe

TDQS

A3.9/5.0

Scored across 2 tools

Disambiguation5/5

The two tools have completely distinct purposes: video_probe inspects metadata, while video_transcribe performs audio transcription. There is no overlap or ambiguity between them.

Naming Consistency5/5

Both tool names follow the same verb_noun snake_case pattern (video_probe, video_transcribe), with the action first and the domain prefix consistent. The naming is uniform and predictable.

Tool Count3/5

With only two tools, the server feels thin for the broad domain of video understanding. The count is borderline but not extreme, as both operations are relevant and non-trivial.

Completeness2/5

The tool surface is severely incomplete for a server named 'video-understanding'. It only covers two basic operations: metadata inspection and speech transcription. There are no tools for common video-understanding tasks such as scene detection, object recognition, action classification, or even audio extraction.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers