Video Understanding MCP
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Video Understanding MCPwatch this video and tell me what's said: https://youtu.be/dQw4w9WgXcQ"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Video Understanding MCP
An MCP server that lets AI agents watch a video and answer questions about it: what is shown, what is said, what text appears on screen, and when. It works with any OpenAI-compatible chat completions endpoint whose model accepts video, such as Gemini models served through a gateway.
Give it a TikTok, YouTube, or X post URL, a direct media URL from a scraper, or a local file, plus a question. It downloads and shrinks the video, sends it to your endpoint, and returns a structured answer:
{
"answer": "The coach demonstrates three squat variations and warns against letting the knees cave in.",
"summary": "A short workout tutorial showing bodyweight squats with form cues.",
"speech": {
"present": true,
"language": "en",
"transcript": "Keep your chest up and push through your heels..."
},
"on_screen_text": ["3 x 12 reps", "Rest 60s"],
"timeline": [{ "start": 12.0, "description": "Switches to jump squats" }],
"analysis": {
"configuredModel": "your-model",
"reportedModel": "gemini-...",
"usage": { "promptTokens": 5400, "completionTokens": 480 },
"sourceDurationSeconds": 58.4,
"analyzedRange": { "start": 0, "end": 58.4 },
"partFormat": "image_url",
"warnings": []
}
}Requirements
Node.js 22.21.0 or newer.
ffmpeg with ffprobe (required). macOS:
brew install ffmpeg. Debian/Ubuntu:sudo apt-get install ffmpeg. Windows:winget install ffmpeg.yt-dlp (optional, needed only for page URLs such as TikTok or YouTube links). macOS:
brew install yt-dlp. Others:pipx install yt-dlp.An OpenAI-compatible endpoint and a model that accepts video input.
Check what the server found with the get_server_info tool.
Related MCP server: avv-mcp
Setup
Add the server to your MCP client. For example, in a JSON-configured client:
{
"mcpServers": {
"video-understanding": {
"command": "npx",
"args": ["-y", "@granitebps/video-understanding-mcp"],
"env": {
"VIDEO_BASE_URL": "https://your-endpoint.example/v1",
"VIDEO_MODEL": "your-video-capable-model",
"VIDEO_API_KEY": "your-endpoint-api-key"
}
}
}
}The server also reads a .env file from its working directory. See .env.example.
Tools
understand_video
Input | Description |
| Required. A page URL (resolved with yt-dlp), a direct media URL, or an absolute local path inside an allowed directory. |
| Optional request headers for a direct media URL, such as |
| Optional. What to find out. Without it you get a general description. |
| Optional range of the source video to analyze. Needed for videos longer than the duration limit; a short range also gives more detail for fast motion such as workouts. |
| Optional frame-sampling hint, applied only by request formats that support it. |
The result fields are shown above. timeline times are seconds from the start of the source video, even when a range was analyzed. analysis.warnings explains anything the server adjusted or could not confirm.
get_server_info
Shows the version, endpoint host and model, request format, limits, local-file policy, and whether ffmpeg, ffprobe, and yt-dlp are available. It never shows the API key.
Request formats
OpenAI-compatible endpoints disagree on how a video is attached to a chat request, and some silently ignore a format they do not understand. Pick the format your endpoint documents with VIDEO_PART_FORMAT:
| Request part | Known to be used by |
|
| OpenRouter, Alibaba Model Studio (Qwen), Tencent TokenHub, vLLM |
|
| OpenRouter; routers that translate image parts by the data URL's MIME type, for example 9router's Gemini path |
|
| LiteLLM (Gemini, vLLM) |
|
| Requesty |
fps is sent only with video_url and file.
If the endpoint ignores the video, the call fails with MEDIA_NOT_PROCESSED instead of returning an answer the model made up. The server detects this by sending the same prompt without the video in parallel (16 output tokens at most) and comparing prompt-token usage, which cancels out any fixed overhead the endpoint adds. This doubles the request count. Turn it off with VIDEO_VERIFY_MEDIA=false only for endpoints that report token usage unusually.
Configuration
Variable | Default | Description |
| — | Required. Endpoint base URL, for example |
| — | Required. Model name or router alias. |
| — | Sent as |
|
|
|
|
| Longest analyzed range (1–7200). |
|
| Largest prepared video sent inline, before base64 (1–100). |
|
| Largest source download (1–10000). |
|
| Endpoint request timeout (5000–900000). |
|
| Analyses running at once; further calls wait (1–16). |
|
|
|
|
| Fail calls when the endpoint evidently ignored the video. |
| — | Directories local files may be read from, separated by |
|
| Allow downloads from loopback and private addresses. |
|
| ffmpeg executable. |
|
| ffprobe executable. |
|
| yt-dlp executable. |
Invalid values stop the server at startup with a message naming the variable. When VIDEO_BASE_URL or VIDEO_MODEL is missing the server still starts, and understand_video returns CONFIG_MISSING.
How a call works
Checks the configuration, that ffmpeg is installed, and that the base URL answers. If it does not, the call fails with
ENDPOINT_UNREACHABLEbefore anything is downloaded.Gets the video: reads an allowed local file, downloads a direct media URL, or resolves a page URL with yt-dlp (refusing over-long videos before downloading).
Cuts the requested range and re-encodes it to H.264 with mono AAC audio, with the shorter side at most 480 pixels (480×854 for a portrait clip), dropping to 360 if needed to fit the upload limit.
Sends it inline as base64 with the question, plus the text-only baseline request.
Validates the reply, drops timeline moments outside the analyzed range, and converts times to source-video seconds.
Temporary files are deleted after every call, including failed and cancelled ones.
Errors
Errors start with a code in brackets, for example [LIMIT_EXCEEDED] The analyzed range is 1500 s, over the 300 s limit ..., and carry the same code in _meta["video-understanding-mcp/error"].
Code | Meaning |
| Bad range or source string. |
|
|
| ffmpeg/ffprobe, or yt-dlp for a page URL, is not installed. |
| The page or media URL could not be fetched or extracted (expired link, unsupported site). |
| Local file outside the allowed directories, or a download target on a private network. |
| The source has no video stream. |
| Over the duration, upload, or download limit, or rejected as too large by the endpoint. |
| The configured base URL is not reachable. |
| The endpoint rejected the API key. |
| The endpoint rate-limited the request; |
| Any other endpoint error, including errors returned with a success status. |
| The endpoint or a media step took too long. |
| The model returned no text. |
| The endpoint evidently ignored the video; try another |
| Unexpected failure. |
Privacy and content rights
The analyzed clip and the question are sent to the endpoint you configure. Use an endpoint whose data handling you accept.
You are responsible for having the right to download and analyze the content you submit, and for following the terms of the sites you download from.
Logs go to stderr as one JSON line per call (code, source host, duration, size, timing, token count). They never contain media, transcripts, answers, URLs with query strings, headers, or the API key.
Security
Downloads refuse loopback, private, link-local, and other non-public addresses, including through redirects, unless
VIDEO_ALLOW_PRIVATE_NETWORK=true. The endpoint itself may be on a local address.Local files are read only from
VIDEO_ALLOWED_DIRS, after resolving symbolic links.Caller-supplied headers go only to the original host and are never passed to yt-dlp.
ffmpeg and ffprobe open inputs as local files only, and playlist or concatenation formats such as HLS are rejected. yt-dlp runs with
--ignore-config.
See SECURITY.md for reporting vulnerabilities.
Known limitations
Videos are sent inline in the request, so each call is bounded by the upload limit. Use
start_secondsandend_secondsfor long videos; the server does not split them automatically.For page URLs, the network restriction checks only the page's host. yt-dlp makes its own follow-up requests to media servers.
When a router falls back between providers, the baseline and the real request can be served by different providers whose fixed overhead differs, so the ignored-video check can misjudge.
analysis.reportedModelshows which model answered.Which upstream provider served a request is not visible beyond the model name the endpoint reports.
Only stdio transport is provided.
Development
npm ci
npm run checknpm run check runs formatting, lint, type checks, tests with coverage, and the package checks. Tests use a fake endpoint, a fake yt-dlp, and ffmpeg-generated clips, so they need ffmpeg but no network or credentials. npm run test:live runs opt-in checks against a real endpoint configured with VIDEO_LIVE_* variables (see .env.example).
See CONTRIBUTING.md and RELEASE.md.
License
Available Tools
2 toolsget_server_infoGet server infoARead-onlyIdempotent
Show the server version, endpoint host and model, request format, limits, local-file policy, and whether ffmpeg and yt-dlp are available. Never shows the API key.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| name | Yes | |
| tools | Yes | |
| limits | Yes | |
| version | Yes | |
| endpoint | Yes | |
| localFiles | Yes | |
| verifyMedia | Yes | |
| dependencies | Yes | |
| allowPrivateNetwork | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, so the safety burden is largely covered; the description nonetheless adds substantive context beyond them by stating the API key is never exposed and by naming the local-file policy. It does not mention cost, latency, or whether the server probe can fail, which keeps it below a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the primary purpose and then the field inventory, closing with the security guarantee. The list is dense but every item is meaningful content the agent would otherwise have to discover by calling; nothing is redundant with the annotations.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the return shape needn't be spelled out here, and the description correctly focuses on scope and the confidentiality caveat. For a zero-parameter diagnostic tool with full annotation coverage, this is essentially complete, lacking only explicit guidance on when an agent should prefer calling it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, which is the baseline-4 case; there is nothing for the description to disambiguate and no schema gap to compensate for. The schema is fully empty and consistent with the description's framing as a no-input inspection call.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
A specific verb ("Show") plus an enumerated inventory of what the response includes: server version, endpoint host, model, request format, limits, local-file policy, and binary availability. An agent knows exactly what this tool reports, and it is unmistakably distinct from the only sibling, understand_video.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: mentioning ffmpeg/yt-dlp availability hints this is a pre-flight diagnostic before video work, and the empty schema signals a safe zero-arg call. However, the description never says "call this first when..." or names any condition or alternative, leaving the agent to infer the trigger.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
understand_videoUnderstand videoARead-only
Watch a video and answer a question about it, using a video-capable model behind the configured OpenAI-compatible endpoint. source accepts a page URL (TikTok, YouTube, X, Instagram, ... resolved with yt-dlp), a direct media URL with optional headers such as Referer from a scraper, or an absolute local path inside an allowed directory. Returns answer, summary, speech (transcript in the original language), on_screen_text, a timeline in source-video seconds, and analysis metadata with warnings. Videos longer than the duration limit need start_seconds/end_seconds; a short range also gives more detail for fast motion such as workouts. fps is a hint applied only when the request format supports it. Errors carry a code (e.g. ENDPOINT_UNREACHABLE, LIMIT_EXCEEDED, SOURCE_NOT_ALLOWED, MEDIA_NOT_PROCESSED) as a [CODE] text prefix and in _meta.
| Name | Required | Description | Default |
|---|---|---|---|
| fps | No | Frame sampling rate hint. Applied only when the endpoint's request format supports it; a short range gives more detail for fast motion | |
| source | Yes | Video to analyze: a page URL (TikTok, YouTube, X, ...), a direct media URL, or an absolute local path inside an allowed directory | |
| headers | No | Request headers for a direct media URL, such as Referer or Cookie returned by a scraper. Sent only to that URL's host | |
| question | No | What to find out about the video. Omit for a general description | |
| end_seconds | No | Analyze up to this second of the source video | |
| start_seconds | No | Analyze from this second of the source video |
Output Schema
| Name | Required | Description |
|---|---|---|
| answer | Yes | |
| speech | Yes | |
| summary | Yes | |
| analysis | Yes | |
| timeline | Yes | |
| on_screen_text | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, openWorld, non-destructive, non-idempotent; the description goes further by disclosing that page URLs are resolved with yt-dlp, that headers are sent only to the target host's host, that fps is applied only when the endpoint's request format supports it, and that failures surface as coded [CODE] prefixes plus _meta. That is substantive behavioral context an agent could not derive from the annotations alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Content is front-loaded with purpose first, then input forms, then return fields, then constraints. It is a single dense block with some run-on clauses (the source-sentence and the error-sentence pack several ideas each), so skimming costs a little, but almost no sentence is filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a six-parameter, open-world media tool with nested headers, the description covers input forms, duration limits, sampling hints, return fields, and an error-code taxonomy. An output schema exists, so the enumerated return values are a bonus rather than a necessity, and nothing an agent needs to invoke it correctly appears to be missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3; the description still adds operational nuance beyond the schema by explaining the scraper-header use case for `headers`, the allowed-directory constraint on local paths, and the conditional application of `fps`. It stops short of adding format/syntax detail for the second-range parameters, which the schema already covers.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a concrete verb+resource: "Watch a video and answer a question about it," and immediately scopes it to a video-capable model behind an OpenAI-compatible endpoint. The accepted `source` forms (page URL, direct media URL, local path) are named, so an agent knows exactly what class of input this tool consumes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives real conditional guidance: long videos require start_seconds/end_seconds, a short range yields more detail for fast motion like workouts, and question may be omitted for a general description. No explicit when-not-to-use or alternative tool routing, but the only sibling is get_server_info, which is not a plausible substitute for this task.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
v1.0.1- First observed
get_server_info - First observed
understand_video
TDQS
Scored across 2 tools
understand_video performs the single domain action while get_server_info is purely diagnostic/introspective. There is no overlap in inputs, outputs, or intent, so an agent cannot plausibly confuse them.
Both names are snake_case verb_noun forms (understand_video, get_server_info) with an imperative verb first. The convention is uniform across the entire surface.
Two tools is on the low end, but the server's scope is narrow — one capability-rich analysis tool plus one configuration/introspection tool. Nothing extraneous is present; it only feels slightly thin because the domain is a single action.
The core lifecycle (analyze a source, inspect server capabilities/limits) is covered, including error codes, timeline output, and truncation handling via start/end seconds. Minor gaps exist around things like probing media metadata without a full analysis or listing allowed directories, but agents can work around these.
Maintenance
Related MCP Connectors
Transcripts, summaries, chapters and timestamped answers for video links, for AI agents.
Video analysis AI: transcripts, summaries, visual scenes/shots, clips, answers in natural language.
Multimodal video analysis MCP — transcription, vision, and OCR for any video URL.
YouTube for AI agents: transcripts, comments, video stats, channel monitor, Shorts captions.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables agents to analyze long videos by downloading them, extracting transcripts and storyboards, and zooming into specific moments with high-resolution frames and OCR.MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to download, convert, and visually analyze videos by providing timestamped frames and locally transcribed speech, all without API keys.MIT
- AlicenseAqualityBmaintenanceEnables AI agents to analyze videos by producing timestamped transcripts, keyframe contact sheets, and metadata reports from local files or online URLs, fully offline and without API keys.6MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to read videos from YouTube links, web pages, media URLs, or local files by returning timestamped transcripts, labelled frame contact sheets, and lossless audio or spectrograms. Also searches YouTube, channels, pages, and folders to find videos worth reading.MIT