Visual Eyes MCP
Connects AI agents to Google's Gemini API for video understanding, enabling natural-language questions, timestamped answers, moment search, and time-range analysis of videos.
Analyzes publicly accessible YouTube videos without downloading them, allowing AI agents to ask questions and search for moments within the video.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Visual Eyes MCP find the moment in this video where the speaker explains the pricing"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Visual Eyes MCP
Give AI agents the ability to see, understand, and search through videos.
Visual Eyes MCP is a Python-based Model Context Protocol (MCP) server that connects AI agents to video understanding capabilities powered by Google's Gemini API.
It allows compatible AI clients to ask questions about videos, find specific moments, and inspect individual frames without manually watching the entire recording.
Status: Work in progress. The project is under active development. No stable release is available yet.
Features
Video understanding: Ask natural-language questions about an entire video.
Timestamped answers: Get approximate timestamps for relevant moments.
Moment search: Find scenes or events using natural-language descriptions.
Time-range analysis: Focus questions on a specific section of a video.
Local video support: Analyze videos stored in explicitly allowed directories.
YouTube support: Analyze publicly accessible YouTube videos without downloading them.
Frame extraction: Extract local frames for closer inspection, including on-screen text.
SQLite caching: Reuse cached analysis results and reduce unnecessary API requests.
Quota-aware requests: Check usage limits before sending requests to Gemini.
Related MCP server: Rustle
How It Works
Visual Eyes MCP acts as a bridge between an MCP-compatible AI client and Google's Gemini API.
You
│
▼
AI Client
│
▼
Visual Eyes MCP Server
│
├── Video source validation
├── Cache and quota checks
├── FFmpeg / FFprobe
├── Gemini API
└── Timestamped results
│
▼
AI ClientThe server handles video validation, processing, caching, and communication with Gemini, then returns structured results to the AI client.
Requirements
Python 3.11 or newer
FFmpeg and FFprobe
A Google Gemini API key
An MCP-compatible client
Installation
Clone the repository:
git clone https://github.com/RMXHM/visual-eyes-mcp.git
cd visual-eyes-mcpInstall dependencies:
uv syncMake sure FFmpeg and FFprobe are available on your system:
ffmpeg -version
ffprobe -versionConfiguration
Visual Eyes MCP reads its Gemini API key from the process environment and from
the project-root .env file. Copy the placeholder file once, then edit it:
cp .env.example .env
# edit .env and replace the placeholder keyThe server and integration test load .env automatically, so you do not need to
export the key for local development. Variables already present in the shell
take precedence over values in .env.
GOOGLE_API_KEY is also supported as a fallback when GEMINI_API_KEY is unset or
blank. GEMINI_API_KEY remains the first choice.
Security notes:
Never commit API keys or other credentials.
Keep local secrets in the ignored
.envfile or your environment..envis ignored by Git;.env.examplecontains placeholders only.API keys are not accepted through the TOML configuration file.
Logs and provider errors are designed to redact credential material.
Additional settings can be configured through the project's TOML configuration and environment overrides. See ARCHITECTURE.md and TECHNICAL_DECISIONS.md.
Development
Run the test suite:
uv run pytestRun lint checks:
uv run ruff check src testsThe ordinary test suite uses fake providers and does not make real Gemini API calls.
Gemini integration test
A separate integration test exercises the real Gemini provider.
It requires two explicit opt-ins:
A non-blank
GEMINI_API_KEYin the process environment or project-root.env.VISUAL_EYES_RUN_INTEGRATION=1in the process environment.
The key is loaded automatically from .env if you have not exported it. The run
flag is read before .env is loaded, so a local .env cannot accidentally turn
an ordinary test run into a paid call.
Run it with:
VISUAL_EYES_RUN_INTEGRATION=1 uv run pytest tests/test_gemini_integration.py -m integration -q -sThe test creates a short local video and makes at most two Gemini generation requests. Provider-level transient retries are disabled for this test.
Files API upload and polling requests are separate from generation requests. Actual API usage may incur charges depending on your Gemini account and model.
The integration model can be overridden using VISUAL_EYES_INTEGRATION_MODEL.
Project Structure
visual-eyes-mcp/
├── src/
│ └── visual_eyes_mcp/
│ ├── analysis.py
│ ├── cache.py
│ ├── config.py
│ ├── errors.py
│ ├── gemini.py
│ ├── media.py
│ ├── models.py
│ ├── prompts.py
│ ├── server.py
│ ├── sources.py
│ └── tools.py
├── tests/
├── ARCHITECTURE.md
├── MVP.md
├── ROADMAP.md
├── TECHNICAL_DECISIONS.md
├── pyproject.toml
└── uv.lockRoadmap
Visual Eyes MCP is still evolving. Planned work and technical decisions are documented in:
Contributing
Contributions, bug reports, and suggestions are welcome, subject to the project's license.
Please open an issue before starting significant changes so proposed work can be discussed.
License
This project is licensed under the PolyForm Noncommercial License 1.0.0.
You may use, study, modify, and share the software for noncommercial purposes under the license terms. Commercial use requires separate permission from the copyright holder.
See LICENSE for the complete terms.
Built by RMXHM
Available Tools
4 toolsanalyze_videoAnswer a question about a videoA
Answer a question about a whole video or a time range, with approximate timestamps. Local files are uploaded to Google for processing; this call is billed to the user's API quota. Use video_info first to check limits.
Inputs: source (required), question (required), detail, refresh, +2 more. Returns: cache, model, usage, answer, +6 more
extract_framesExtract video frames locallyB
Extract still images from a local video at timestamps or a fixed interval and return them as JPEG content. Runs ffmpeg locally: no upload and no API quota.
Inputs: source (required), width, max_frames, timestamps, +3 more.
find_momentsLocate moments in a videoC
Find ranked segments matching a description. Local files are uploaded to Google for processing; this call is billed to the user's API quota.
Inputs: source (required), description (required), detail, refresh, +3 more. Returns: cache, model, usage, detail, +3 more
video_infoInspect a video sourceA
Report duration, dimensions, codec facts, cache statistics, and the configured limits for a local video or public YouTube URL. No model call and no upload.
Inputs: source (required). Returns: fps, notes, width, height, +8 more
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
The four tools split cleanly along output type: raw JPEG frames (extract_frames), metadata/limits (video_info), natural-language answers (analyze_video), and ranked time segments (find_moments). The main residual overlap is between analyze_video answering a time-range question and find_moments returning matching segments, but the descriptions make the distinction clear enough to choose correctly.
All names are snake_case and three follow a verb_noun pattern (extract_frames, analyze_video, find_moments). video_info deviates by being noun-based rather than verb-led (compare get_video_info), a minor inconsistency that is still perfectly readable.
Four tools for a focused video-inspection server is tight and each has a distinct role, with no filler. It sits at the low end of the comfortable range, but nothing feels redundant or missing at the tool-count level.
The surface covers the full lifecycle of inspecting a video: probe limits, extract frames, ask questions, and search for moments, with sensible guidance to call video_info first. Minor gaps remain (no audio/transcript extraction, no clip export, and frame extraction is local-only while analysis supports YouTube URLs).
Maintenance
Related MCP Connectors
YouTube for AI agents: transcripts, comments, video stats, channel monitor, Shorts captions.
Transcripts, summaries, chapters and timestamped answers for video links, for AI agents.
YouTube transcripts, search, channel/playlist listings and upload tracking for AI agents.
Search YouTube videos, read transcripts and get public video information in your AI assistant.
Related MCP Servers
- AlicenseAqualityBmaintenanceEnables AI agents to search videos, retrieve metadata and transcripts, pinpoint exact moments via hybrid semantic search, and conduct multi-video research with timestamp deep links—all without YouTube API keys.51MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to read videos from YouTube links, web pages, media URLs, or local files by returning timestamped transcripts, labelled frame contact sheets, and lossless audio or spectrograms. Also searches YouTube, channels, pages, and folders to find videos worth reading.MIT
- AlicenseNot gradedqualityCmaintenanceLets agents watch public YouTube videos and answer questions about their audio and visuals, using Gemini for transcript-aware or fixed-frame analysis.1MIT
- AlicenseNot gradedqualityBmaintenanceEnables an agent to process local video files and then read, search and inspect the results: timestamped transcripts, on-screen text and key frames as images. Everything runs locally, so an agent can answer questions like how to perform a task shown in a screencast or meeting recording.379 npm1MIT