VideoContext MCP

VideoContext MCP is a local-first Python MCP server that helps compatible AI clients understand YouTube and local video through metadata, timestamped transcripts, unified timelines, and selected visual frames. It prepares media locally and exposes bounded evidence over stdio; it does not automatically send an entire raw video to the connected model.

The Python package is video_understanding_mcp, the command is video-mcp, and the local cache application identifier is video-understanding-mcp.

Why it exists

Long videos are usually too large to inspect reliably as one model input. VideoContext MCP performs acquisition, transcription, range selection, frame sampling, and artifact management on the local machine. The client can then read only the transcript ranges and images needed for a question.

This makes the workflow inspectable and controllable: transcript text is timestamped, frame selection is reproducible, preview images come before full-frame retrieval, and completed jobs can be expired or deleted. Actual model-context use still depends on the client and on what it chooses to retrieve.

Related MCP server: MCP YouTube Transcript Pro

Project status

Version 0.1.0 is the stable initial release. The implemented local and YouTube workflows, strict public schemas, caption and local faster-whisper policies, half-open frame ranges, atomic publication, bounded retrieval, and cleanup lifecycle are covered by deterministic tests. Bounded live acceptance checks have also exercised acquisition, medium-model fallback, transcript/timeline retrieval, preview-first inspection, selected frame retrieval, and deletion.

“Stable” here means the documented initial feature set is working and release-tested. It is not a production certification or a guarantee that every YouTube extractor response, codec, client, or machine behaves identically.

Features

• Inspect one canonical YouTube video without downloading its media.

• Inspect an allowlisted local video using a fast snapshot or hardened private copy.

• Prepare a whole video or a bounded half-open time range.

• Prefer permitted YouTube captions and fall back to an explicitly installed local Whisper model.

• Produce WebVTT transcripts, compact chronological timelines, frame manifests, and metadata.

• Select frames by automatic profile, interval, FPS, scene change, hybrid sampling, or timestamps.

• Retrieve small previews first, then only explicitly selected full stored frames.

• Delete jobs explicitly or expire them using normal or ephemeral retention.

• Enforce strict schemas, filesystem boundaries, queues, timeouts, and resource ceilings.

• Run diagnostics through both the CLI and the public MCP tool.

Architecture

The intended client workflow is:

inspect video
→ choose a useful range and transcript settings
→ prepare locally
→ read transcript and timeline
→ inspect frame previews
→ retrieve selected full frames
→ delete the job

One local stdio server owns two source-specific trust boundaries and one shared processing pipeline:

MCP client
  |-- inspect_youtube_video --> exact HTTPS YouTube validation --> yt-dlp metadata worker
  |-- inspect_local_video   --> allowed-directory validation --> fast snapshot or hardened copy
  |-- prepare_youtube_video --> bounded yt-dlp worker ---------+
  `-- prepare_local_video   --> secure revalidation + copy ----+--> shared pipeline
                                                               |    probe -> plan -> audio/
                                                               |    captions -> frames ->
                                                               |    deduplicate/prioritize ->
                                                               |    atomic private publication
                                                               `--> compact resources
                                                                    + selective previews/frames

There is no HTTP listener, remote upload route, LLM invocation, browser automation, shell tool, or user-controlled output path. See docs/ARCHITECTURE.md for the detailed planning and lifecycle algorithms.

Requirements

• Python 3.11, 3.12, or 3.13; Python 3.12 is used by the lock file verification environment.

• FFmpeg and FFprobe 6 or newer, installed as system executables.

• uv for the locked installation workflow, or another Python installer that honors pyproject.toml.

• Local storage for the chosen Whisper model, selected source range, frames, and cache reserve. The default job policy reserves 5 GiB of free space plus its conservative work estimate.

The deterministic CI matrix runs on current Ubuntu, macOS, and Windows images. Local live media and Whisper acceptance testing has been performed on macOS. Production stdio deliberately refuses root/Administrator execution; non-Administrator Windows production startup has not been exercised.

Install FFmpeg/FFprobe first:

# macOS (Homebrew)
brew install ffmpeg

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install --yes ffmpeg python3-venv

# Windows (PowerShell with Chocolatey)
choco install ffmpeg python312

Installation

Clone the repository and install the locked runtime:

git clone https://github.com/sakshamjain2301/videocontext-mcp.git
cd videocontext-mcp
uv sync --frozen

For development and the complete verification toolchain:

uv sync --frozen --extra dev

The repository name does not change the video-mcp command, the video_understanding_mcp package, or the video-understanding-mcp cache identifier.

On Windows, replace .venv/bin/python and .venv/bin/video-mcp with .venv\Scripts\python.exe and .venv\Scripts\video-mcp.exe.

No API key or .env file is required. The project intentionally does not install FFmpeg or other system software itself.

Whisper model installation

The transcription backend is faster-whisper, which runs local CTranslate2 models. MCP requests never download a model. Install each desired model explicitly from the administrator CLI:

.venv/bin/video-mcp install-model base
.venv/bin/video-mcp install-model small
.venv/bin/video-mcp install-model medium

Approved names are tiny, tiny.en, base, base.en, small, small.en, medium, medium.en, and large-v3. The default is base. Smaller models generally require less local compute; larger models generally trade more compute and storage for transcription capability. No project-specific benchmark is claimed.

The .en variants are English-only and cannot be used for translation. The multilingual models support transcription and, where the backend permits it, translation. This project does not implement a whisper.cpp backend; whisper.cpp model files are not compatible with this installer.

To keep models in a chosen private location:

.venv/bin/video-mcp install-model base --model-cache-dir /path/to/private/model-cache

If a requested model is absent, frame preparation can return partial_success with an actionable warning instead of silently downloading it. English-only .en models cannot translate; translate is allowed only for compatible multilingual models.

By default models are stored under the platform-specific video-understanding-mcp user-cache directory in models/<model-name>. Use --model-cache-dir to select another private administrator-owned location. video-mcp doctor lists which approved models are detected; it does not download or load them for inference.

Doctor command

Run before configuring a client:

.venv/bin/video-mcp doctor --allow-dir /path/to/allowed-videos

The report covers Python, MCP SDK, FFmpeg, FFprobe, yt-dlp, faster-whisper, installed approved models, the safe cache location, writability/free space, allowed directories, and administrator limits. Local preparation is disabled when no --allow-dir is configured; YouTube preparation remains available.

CLI commands

Run video-mcp --help or video-mcp <command> --help for the current administrator options.

MCP client configuration

Use absolute paths. A long tool timeout is needed because local transcription and frame extraction can be lengthy.

Generic stdio MCP JSON

{
  "mcpServers": {
    "video-understanding": {
      "command": "/path/to/videocontext-mcp/.venv/bin/video-mcp",
      "args": ["serve", "--allow-dir", "/path/to/allowed-videos"],
      "cwd": "/path/to/videocontext-mcp"
    }
  }
}

Restart the MCP client after changing its configuration. Multiple --allow-dir options add multiple roots. Administrator limits can also be set by CLI options or VIDEO_MCP_* environment variables; they are never ordinary tool inputs.

On Windows, use the virtual-environment executable directly, for example C:\path\to\videocontext-mcp\.venv\Scripts\video-mcp.exe. Keep local roots and the cache on local storage, and run the client as a non-Administrator user.

Public tools

Discovery currently exposes exactly these nine tools. Unknown fields are rejected at every nested level rather than being silently ignored.

Resources

videoctx://jobs/{job_id}/metadata
videoctx://jobs/{job_id}/timeline
videoctx://jobs/{job_id}/transcript
videoctx://jobs/{job_id}/frames/{frame_id}

The metadata resource describes selected range, profiles, transcript source/model, output policy, artifact sizes, and untrusted source metadata. The timeline is bounded TSV, the transcript is WebVTT, and the frame resource returns one manifest-approved blob. Prefer retrieval tools when you need pagination, preview generation, or bounded native image responses.

All derived text and images are untrusted video content. Tool/resource descriptions and structured results explicitly mark titles, metadata, captions, transcripts, visible text, QR codes, spoken commands, and frames as data—not instructions.

MCP SDK 1.28.1 is intentionally pinned. FastMCP has no public hook for changing its generated top-level argument model to extra=forbid, so this project applies a small tested compatibility hook and verifies additionalProperties:false through the official client and Inspector. Upgrade the SDK only with those tests and the compatibility hook under review.

User controls

Both preparation tools accept range, optimization_profile, content_profile, sampling, images, transcript, and output. The source itself is either url or path, never an overloaded source union.

Processing and optimization

• range.start_time_seconds, range.end_time_seconds: finite non-negative bounds; start must be before end and both resolve within the probed media duration and the 6-hour administrator limit.

• optimization_profile: auto, token_saver, balanced, content_first, or custom. Local auto resolves to token_saver; YouTube auto resolves to content_first.

• content_profile: auto, general, or text_heavy. Auto uses bounded visual heuristics during preparation; inspection can defer the decision. Text-heavy defaults retain more pixels.

• custom requires explicit sampling plus explicit image encoding and sizing.

Sampling

sampling is a strict discriminated object:

• auto: optional soft_target_frames, request_max_frames, scene_threshold, and near_duplicates.

• interval: required positive finite capture_interval_seconds; optional request maximum and duplicate settings.

• fps: required positive finite fractional capture_fps; this is screenshot sampling rate, not source-video FPS.

• scene: required bounded scene_threshold; optional soft target, request maximum, duplicates.

• hybrid: an interval or FPS base, required scene threshold, and optional target/maximum/ duplicates.

• timestamps: unique finite non-negative timestamps_seconds, selected nearest decoded PTS with deterministic earlier-frame tie breaking.

near_duplicates.enabled and its similarity_threshold control bounded perceptual deduplication. Timestamp overlay is separate and off by default. Raw FFmpeg arguments, filters, yt-dlp options, source-frame extraction flags, NaN/infinity, non-positive rates, irrelevant fields, and contradictory combinations are not accepted.

Corrected local token-saver default

The local base-frame interval has no 30-second maximum:

local_interval = max(selected_duration / 90, 3 seconds)

This directly produces approximately 90 evenly distributed base candidates beyond about 4.5 minutes: about 3.33 seconds for 5 minutes, 20 seconds for 30 minutes, 40 seconds for 1 hour, 80 seconds for 2 hours, and 240 seconds for 6 hours. Very short videos retain the 3-second minimum.

After base extraction, only major scenes (conservative 0.45 threshold) and useful chapter boundaries supplement coverage. Aggressive near-duplicate removal and nearby-base rejection are applied. The final soft target remains about 120 and the automatic request ceiling remains 180; unused capacity is never filled with low-value frames. The non-bypassable administrator ceiling is 3,000.

Choose balanced, content_first, an interval/FPS, scene/hybrid, explicit timestamps, or higher soft/request settings (within administrator limits) for denser local coverage.

YouTube content-first auto uses an interval clamped to 2–10 seconds and a qualified scene-addition budget of 35%, rising to 50% only for high-dynamic content. It also does not fill unused capacity with low-value candidates.

Images

• encoding: jpeg quality 1–95 (default), png compression 0–9, or webp quality/lossless. WebP is rejected unless the administrator explicitly enables verified client compatibility.

• sizing: original, bounded fit maximum width/height, or bounded exact width/height.

• Aspect ratio is preserved by default. exact.stretch=true is the explicit distortion opt-in.

• allow_upscale defaults false. Width, height, total pixels, and completed bytes remain bounded.

• timestamp_overlay defaults false.

• contact_sheets optionally controls 2–10 columns, 1–100 frames per sheet, and labels. Individual frame files remain available so contact sheets are never the only small-text representation.

Adaptive defaults are JPEG: local token-saver general 960×540 quality 80; token-saver text-heavy 1280×720 quality 85; balanced/content-first general 1280×720 quality 85; text-heavy 1920×1080 quality 90. Frames are not upscaled. Frame count and pixel dimensions generally matter more to model context than JPEG versus PNG.

Transcript policies

Local videos use the common Whisper settings directly because they have no YouTube caption source. The common fields are model (default base), language (auto or a validated language tag), task (transcribe or compatible translate), segment_timestamps (required true), and optional word_timestamps (default false).

Creator captions and automatic captions remain distinct in metadata. Automatic captions are not accepted by creator_captions_then_whisper; the server does not silently substitute them. Whisper metadata records the selected local model. Caption transcripts record no Whisper model.

No audio produces a successful frames-only result with a warning. No speech produces an empty transcript status instead of a crash. If transcription is unavailable after frames succeed, the result accurately reports partial success.

Output and retention

• generate_context_markdown and generate_timeline_jsonl are optional and off by default.

• ephemeral selects a 15-minute completed-output TTL; normal TTL is 24 hours.

• source_retention is delete_after_processing by default or retain_until_job_expiry. The private local copy/download is deleted only after successful atomic publication when using the default. The original local source is never changed.

• Contact sheets are controlled under image settings. transcript.vtt, metadata.json, compact timeline.tsv, frame manifest, and individual frames are canonical outputs.

Example workflow

The following illustrates the public MCP sequence. Replace the URL, job ID, and frame IDs with values returned by your own calls.

1. Inspect before downloading media:

{
  "url": "https://www.youtube.com/watch?v=VIDEO_ID",
  "optimization_profile": "auto",
  "content_profile": "auto"
}

2. Prepare only the useful ten-minute range, preferring creator captions and falling back to the installed base model:

{
  "url": "https://www.youtube.com/watch?v=VIDEO_ID",
  "range": {
    "start_time_seconds": 600,
    "end_time_seconds": 1200
  },
  "optimization_profile": "content_first",
  "content_profile": "text_heavy",
  "transcript": {
    "policy": "creator_captions_then_whisper",
    "model": "base",
    "language": "auto",
    "task": "transcribe",
    "segment_timestamps": true,
    "word_timestamps": false
  },
  "output": {
    "source_retention": "delete_after_processing",
    "ephemeral": false
  }
}

3. Read the returned job timeline, narrowing by milliseconds or following a continuation token when the response is truncated:

{"job_id":"<job-id>","start_ms":600000,"end_ms":1200000,"max_chars":20000}

4. Request previews for the relevant interval:

{"job_id":"<job-id>","time_range":{"start_ms":600000,"end_ms":720000},"max_previews":8,"preview_width":640}

5. Retrieve only the one or two frames needed for fine text or visual details:

{"job_id":"<job-id>","frame_ids":["f_000001","f_000004"],"max_frames":4}

6. Delete the job when it is no longer needed:

{"job_id":"<job-id>"}

Additional processing examples

Inspection before approval:

{"url":"https://www.youtube.com/watch?v=VIDEO_ID","optimization_profile":"auto"}

Local defaults (token-saver):

{"path":"/path/to/allowed-videos/demo.mp4"}

YouTube defaults (content-first):

{"url":"https://youtu.be/VIDEO_ID"}

Interval, restricted to 10–20 minutes:

{
  "path":"/path/to/allowed-videos/lecture.mp4",
  "range":{"start_time_seconds":600,"end_time_seconds":1200},
  "sampling":{"mode":"interval","capture_interval_seconds":5,"request_max_frames":120}
}

Fractional screenshot FPS:

{"path":"/path/to/allowed-videos/demo.mp4","sampling":{"mode":"fps","capture_fps":0.5}}

Scene mode:

{"path":"/path/to/allowed-videos/demo.mp4","sampling":{"mode":"scene","scene_threshold":0.35}}

Hybrid interval plus scenes:

{
  "path":"/path/to/allowed-videos/demo.mp4",
  "sampling":{"mode":"hybrid","base":{"mode":"interval","capture_interval_seconds":10},"scene_threshold":0.35}
}

Exact timestamps and text-heavy PNG output:

{
  "path":"/path/to/allowed-videos/slides.mp4",
  "content_profile":"text_heavy",
  "sampling":{"mode":"timestamps","timestamps_seconds":[0,12.5,61]},
  "images":{"encoding":{"format":"png","compression_level":6},"sizing":{"mode":"fit","max_width":1920,"max_height":1080}}
}

Hybrid FPS uses {"base":{"mode":"fps","capture_fps":0.25}}. A deliberately dense custom configuration must include optimization_profile:"custom", explicit sampling, encoding, and sizing.

Preview-first retrieval:

{"job_id":"<32-character-job-id>","time_range":{"start_ms":60000,"end_ms":120000},"max_previews":8,"preview_width":640}

Then retrieve only selected full frames:

{"job_id":"<32-character-job-id>","frame_ids":["f_000001","f_000004"],"max_frames":4}

Performance guidance

• Prepare the smallest useful half-open range; Whisper and frame extraction work only on that selected interval.

• Larger Whisper models can consume substantially more CPU, memory, disk, and wall time. CPU-only transcription can be slower than real time.

• Start with timeline text and low-resolution previews. Full-frame retrieval is intended for a few diagrams, formulas, slides, or code views—not bulk transfer.

• “Full frame” means the complete stored frame. Its resolution cannot exceed the progressive source selected by the secure downloader, and frames are not upscaled by default.

• Automatic planners target useful coverage rather than filling the caller's capacity with low-value or duplicate images.

Output and model-context behavior

timeline.tsv uses integer milliseconds and one chronological event per row. Transcript tabs and newlines become spaces while Unicode is preserved. Frame rows reference IDs/resource URIs; images are never placed as base64 inside Markdown, TSV, JSON, or ordinary result text. MCP-native image blocks are returned only by bounded retrieval calls.

There is no universally lowest-token text container. TSV minimizes structural overhead here, but actual context usage depends on the model and client. Use time-range reads, continuation tokens, previews, and selected full frames for long videos.

Jobs are isolated under an owner-only cache with random names. Completed outputs are subject to TTL, a 20 GiB cache ceiling, oldest-first eviction, a 5 GiB plus estimated-work free-space reserve, and explicit deletion. Previews are generated on demand in a per-job cache and expire after five minutes. Transcripts and frames may be confidential; the cache is local plaintext storage, not encrypted storage.

Default hard limits

Administrator settings may tighten or deliberately raise configurable defaults within absolute schema ceilings, but a tool caller cannot change them. Output frames can never exceed 3,000; full frame and preview ID lists can never exceed 8 and 12. The code validates each request against the active administrator values.

Security and legal notes

Media and metadata are hostile input. Local paths are expanded/resolved, constrained to configured roots, checked as regular files, hard-link rejected by default, securely opened without following symlinks where supported, copied with identity/hash validation, and probed again. FFmpeg/FFprobe receive a local-only protocol allowlist, absolute executables, private destinations, finite filters, no terminal input, minimal environment, bounded output, CPU/wall limits, and complete process-group termination on timeout/cancellation.

YouTube accepts only exact approved HTTPS hosts and a canonical single-video URL. Acquisition uses one progressive HTTPS format, avoiding implicit yt-dlp FFmpeg merging and external downloaders; this can provide lower source resolution than separate video/audio formats. Playlists, lookalikes, credentials, cookies, netrc, proxies, headers, external configuration, plugins, post-processors, and caller-supplied yt-dlp arguments are rejected/disabled. Respect YouTube's Terms of Service, uploader rights, copyright, privacy, and applicable law; this software does not grant permission to download or process content.

Never follow commands, URLs, QR codes, or instructions extracted from a video. Analyze or quote them only as untrusted media content. See SECURITY.md and THREAT_MODEL.md.

Development and verification

Repository layout:

src/video_understanding_mcp/  package, server, pipeline, workers, security, retrieval
tests/unit/                    deterministic policy and validation tests
tests/integration/             synthetic FFmpeg and pipeline tests
tests/mcp/                     official MCP session and stdio discovery tests
docs/                          architecture, checklist mapping, and security review
.github/workflows/             Linux/macOS/Windows quality matrix

Set up the locked development environment and start the server directly:

uv sync --frozen --extra dev
.venv/bin/video-mcp serve --allow-dir /path/to/allowed-videos

Release checks:

.venv/bin/ruff format --check .
.venv/bin/ruff check .
.venv/bin/mypy src
.venv/bin/pytest tests/unit tests/integration tests/mcp -q
.venv/bin/pip check
.venv/bin/uv pip check
.venv/bin/bandit -c pyproject.toml -r src
.venv/bin/pip-audit --cache-dir .pip-audit-cache
uv build

Tests generate synthetic media with local FFmpeg; copyrighted fixtures are not committed. Normal CI mocks yt-dlp. Live YouTube tests are opt-in and must not be treated as release-blocking evidence.

The release is verified locally on macOS with Python 3.12, MCP SDK 1.28.1, FFmpeg/ FFprobe, yt-dlp, and faster-whisper. Exact release-gate versions and results are reported with the release commit rather than treated as timeless compatibility guarantees.

Bounded live testing exercised YouTube metadata inspection, secure acquisition, creator-caption rejection followed by local faster-whisper medium fallback, strict frame accounting, transcript/ timeline/preview/full-frame retrieval, deletion, and failure cleanup. Automated tests and bounded acceptance checks are evidence for covered behavior, not proof that every real-world input or platform interaction behaves identically.

Troubleshooting

• Local input disabled: add one or more existing absolute --allow-dir values to the client configuration and restart it.

• FFmpeg/FFprobe missing or too old: install/update the system package, then rerun doctor.

• Whisper model not installed: run the explicit install-model command outside an MCP request.

• Whisper model rejected: choose one of the approved names listed above; arbitrary filesystem models and whisper.cpp files are not accepted.

• No creator captions: use the default creator-captions-then-Whisper policy with an installed local model, or deliberately select another documented policy.

• Tool timeout at 60 seconds: set the client's tool_timeout_sec to a suitable bounded value, such as 28,800 seconds for the server's default maximum wall time.

• Insufficient disk: free space, shorten the range, reduce frames/resolution, or use a separate private --cache-dir; the server will not weaken its reserve to continue.

• WebP disabled: use JPEG/PNG or explicitly enable WebP only after validating the complete MCP/client image path.

• Partial transcript: inspect warnings; frames remain available when no audio/speech/model or a permitted caption source is unavailable.

• Resource expired/not found: completed jobs expire and may be evicted; prepare again if needed.

• YouTube rejected: use one canonical video URL on an approved exact hostname, without playlist parameters.

• Playlist rejected: pass a clean single-video URL; playlist expansion is intentionally disabled.

• Wrong worker interpreter: configure the MCP client with the absolute video-mcp executable from this project's virtual environment. Do not point it at an unrelated base Python.

• yt-dlp unavailable or outdated: rerun uv sync --frozen, then use video-mcp doctor to verify the installed dependency before inspecting YouTube.

Current limitations

• Local faster-whisper is the only transcription backend; models must be installed explicitly.

• CPU-only Whisper can be slower than the selected source range, especially for larger models.

• Public YouTube inspection reports chapter availability but does not currently expose chapter boundary timestamps for range selection.

• Secure YouTube acquisition deliberately selects one progressive HTTPS audio/video format. Its resolution may be lower than separate adaptive streams.

• Preparation and retrieval concurrency are intentionally bounded for one local user rather than optimized as a multi-user service.

• Windows CI covers deterministic behavior, but non-Administrator production stdio startup and complete Windows Job Object accounting for very short-lived child CPU remain unverified.

License and release status

VideoContext MCP is licensed under the MIT License; see LICENSE.

Version 0.1.0 is the stable initial release for the documented local-first feature set. It should not be exposed as a network service or described as universally production-certified.