Skip to main content
Glama

Visual Eyes MCP

Give AI agents the ability to see, understand, and search through videos.

Visual Eyes MCP is a Python-based Model Context Protocol (MCP) server that connects AI agents to video understanding capabilities powered by Google's Gemini API.

It allows compatible AI clients to ask questions about videos, find specific moments, and inspect individual frames without manually watching the entire recording.

Status: Work in progress. The project is under active development. No stable release is available yet.

Features

  • Video understanding: Ask natural-language questions about an entire video.

  • Timestamped answers: Get approximate timestamps for relevant moments.

  • Moment search: Find scenes or events using natural-language descriptions.

  • Time-range analysis: Focus questions on a specific section of a video.

  • Local video support: Analyze videos stored in explicitly allowed directories.

  • YouTube support: Analyze publicly accessible YouTube videos without downloading them.

  • Frame extraction: Extract local frames for closer inspection, including on-screen text.

  • SQLite caching: Reuse cached analysis results and reduce unnecessary API requests.

  • Quota-aware requests: Check usage limits before sending requests to Gemini.

Related MCP server: Rustle

How It Works

Visual Eyes MCP acts as a bridge between an MCP-compatible AI client and Google's Gemini API.

You
 │
 ▼
AI Client
 │
 ▼
Visual Eyes MCP Server
 │
 ├── Video source validation
 ├── Cache and quota checks
 ├── FFmpeg / FFprobe
 ├── Gemini API
 └── Timestamped results
 │
 ▼
AI Client

The server handles video validation, processing, caching, and communication with Gemini, then returns structured results to the AI client.

Requirements

  • Python 3.11 or newer

  • uv

  • FFmpeg and FFprobe

  • A Google Gemini API key

  • An MCP-compatible client

Installation

Clone the repository:

git clone https://github.com/RMXHM/visual-eyes-mcp.git
cd visual-eyes-mcp

Install dependencies:

uv sync

Make sure FFmpeg and FFprobe are available on your system:

ffmpeg -version
ffprobe -version

Configuration

Visual Eyes MCP reads its Gemini API key from the process environment and from the project-root .env file. Copy the placeholder file once, then edit it:

cp .env.example .env
# edit .env and replace the placeholder key

The server and integration test load .env automatically, so you do not need to export the key for local development. Variables already present in the shell take precedence over values in .env.

GOOGLE_API_KEY is also supported as a fallback when GEMINI_API_KEY is unset or blank. GEMINI_API_KEY remains the first choice.

Security notes:

  • Never commit API keys or other credentials.

  • Keep local secrets in the ignored .env file or your environment.

  • .env is ignored by Git; .env.example contains placeholders only.

  • API keys are not accepted through the TOML configuration file.

  • Logs and provider errors are designed to redact credential material.

Additional settings can be configured through the project's TOML configuration and environment overrides. See ARCHITECTURE.md and TECHNICAL_DECISIONS.md.

Development

Run the test suite:

uv run pytest

Run lint checks:

uv run ruff check src tests

The ordinary test suite uses fake providers and does not make real Gemini API calls.

Gemini integration test

A separate integration test exercises the real Gemini provider.

It requires two explicit opt-ins:

  1. A non-blank GEMINI_API_KEY in the process environment or project-root .env.

  2. VISUAL_EYES_RUN_INTEGRATION=1 in the process environment.

The key is loaded automatically from .env if you have not exported it. The run flag is read before .env is loaded, so a local .env cannot accidentally turn an ordinary test run into a paid call.

Run it with:

VISUAL_EYES_RUN_INTEGRATION=1 uv run pytest tests/test_gemini_integration.py -m integration -q -s

The test creates a short local video and makes at most two Gemini generation requests. Provider-level transient retries are disabled for this test.

Files API upload and polling requests are separate from generation requests. Actual API usage may incur charges depending on your Gemini account and model.

The integration model can be overridden using VISUAL_EYES_INTEGRATION_MODEL.

Project Structure

visual-eyes-mcp/
├── src/
│   └── visual_eyes_mcp/
│       ├── analysis.py
│       ├── cache.py
│       ├── config.py
│       ├── errors.py
│       ├── gemini.py
│       ├── media.py
│       ├── models.py
│       ├── prompts.py
│       ├── server.py
│       ├── sources.py
│       └── tools.py
├── tests/
├── ARCHITECTURE.md
├── MVP.md
├── ROADMAP.md
├── TECHNICAL_DECISIONS.md
├── pyproject.toml
└── uv.lock

Roadmap

Visual Eyes MCP is still evolving. Planned work and technical decisions are documented in:

Contributing

Contributions, bug reports, and suggestions are welcome, subject to the project's license.

Please open an issue before starting significant changes so proposed work can be discussed.

License

This project is licensed under the PolyForm Noncommercial License 1.0.0.

You may use, study, modify, and share the software for noncommercial purposes under the license terms. Commercial use requires separate permission from the copyright holder.

See LICENSE for the complete terms.


Built by RMXHM

Available Tools

4 tools
analyze_videoAnswer a question about a videoA

Answer a question about a whole video or a time range, with approximate timestamps. Local files are uploaded to Google for processing; this call is billed to the user's API quota. Use video_info first to check limits.

Inputs: source (required), question (required), detail, refresh, +2 more. Returns: cache, model, usage, answer, +6 more

extract_framesExtract video frames locallyB

Extract still images from a local video at timestamps or a fixed interval and return them as JPEG content. Runs ffmpeg locally: no upload and no API quota.

Inputs: source (required), width, max_frames, timestamps, +3 more.

find_momentsLocate moments in a videoC

Find ranked segments matching a description. Local files are uploaded to Google for processing; this call is billed to the user's API quota.

Inputs: source (required), description (required), detail, refresh, +3 more. Returns: cache, model, usage, detail, +3 more

video_infoInspect a video sourceA

Report duration, dimensions, codec facts, cache statistics, and the configured limits for a local video or public YouTube URL. No model call and no upload.

Inputs: source (required). Returns: fps, notes, width, height, +8 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.4/5.0

    Scored across 4 tools

    Disambiguation4/5

    The four tools split cleanly along output type: raw JPEG frames (extract_frames), metadata/limits (video_info), natural-language answers (analyze_video), and ranked time segments (find_moments). The main residual overlap is between analyze_video answering a time-range question and find_moments returning matching segments, but the descriptions make the distinction clear enough to choose correctly.

    Naming Consistency4/5

    All names are snake_case and three follow a verb_noun pattern (extract_frames, analyze_video, find_moments). video_info deviates by being noun-based rather than verb-led (compare get_video_info), a minor inconsistency that is still perfectly readable.

    Tool Count4/5

    Four tools for a focused video-inspection server is tight and each has a distinct role, with no filler. It sits at the low end of the comfortable range, but nothing feels redundant or missing at the tool-count level.

    Completeness4/5

    The surface covers the full lifecycle of inspecting a video: probe limits, extract frames, ask questions, and search for moments, with sensible guidance to call video_info first. Minor gaps remain (no audio/transcript extraction, no clip export, and frame extraction is local-only while analysis supports YouTube URLs).

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to read videos from YouTube links, web pages, media URLs, or local files by returning timestamped transcripts, labelled frame contact sheets, and lossless audio or spectrograms. Also searches YouTube, channels, pages, and folders to find videos worth reading.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables an agent to process local video files and then read, search and inspect the results: timestamped transcripts, on-screen text and key frames as images. Everything runs locally, so an agent can answer questions like how to perform a task shown in a screencast or meeting recording.
      379 npm
      1
      MIT