video-to-context
Enables analysis of YouTube videos by URL, including downloading the video, transcribing speech, extracting on-screen text and visual context, and producing timestamped Markdown/JSON records.
Supports processing YouTube Shorts links, turning short-form videos into structured timestamped text with audio, on-screen text, and visual descriptions for use by AI tools.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Generate Text Contexts from Youtube Shorts
Turn a short-form Youtube video into structured, timestamped context in a text format. It includes both audio and visual context.
Most language models can't watch a video. This tool gives them a faithful text version of one: what was said, what was written on screen, and what was shown, with every item labelled by source and time.
[ON-SCREEN:LOGO 00:00–00:16] EXAMPLE CHANNEL
[AUDIO 00:07–00:11] Today we're testing three ways to brew coffee at home.
[ON-SCREEN 00:08–00:10] 3 WAYS TO BREW COFFEE
[VISUAL 00:07–00:16] A person in an apron pours hot water into a glass coffee maker in a bright kitchen.Use the output as the input to summarisation, search, fact-checking, content moderation, RAG, or your own agents.
Contents
Related MCP server: youtube-mcp
How it works
YouTube link or video file
│
download and check
│
├── audio ──► speech-to-text (Whisper, runs locally)
│
└── frames ─► vision model: on-screen text, scenes, settings, editing
│
record.json + record.mdQuick start
1. Install the requirements
You need uv, ffmpeg, and about 5 GB of free disk space. uv installs the right Python version for you.
System | Install command |
macOS |
|
Ubuntu or Debian |
|
2. Install video-to-context
uv tool install --python 3.12 video-to-contextAdd uv's tools to your PATH, then open a new terminal:
uv tool update-shellYou can also use pip install video-to-context inside a Python 3.12 environment.
3. Choose a vision model (optional)
Download the settings template:
mkdir -p ~/.video-to-context
curl -o ~/.video-to-context/.env https://raw.githubusercontent.com/wani-ham/youtube-shorts-to-text-context/main/.env.exampleOpen ~/.video-to-context/.env and uncomment one option from the next section. Without one, the tool still produces the transcript and reads on-screen text locally, but it won't include visual analysis.
4. Run it
video-to-context "https://www.youtube.com/shorts/VIDEO_ID"The first run is slower because it downloads the speech and OCR models. When it finishes, it prints the path to your result in a store/ folder inside your current directory.
Choosing a vision model
You don't need Claude. Any service with an OpenAI-compatible API and image support will work.
Option | Cost | Your frames go to | Best for |
Claude Haiku 5.5 | About 1 cent per video (estimate) | Anthropic | Tested default |
Gemini | Free tier available | Trying it at no cost | |
OpenAI | Paid | OpenAI | Existing OpenAI users |
OpenRouter | Free and paid models | The selected provider | Comparing models |
Ollama | Free | Nowhere, runs locally | Private text extraction, no account |
None | Free | Nowhere | Transcript and text only |
Status: Claude has been tested on real videos. Ollama with Gemma 4 12B has been tested on one real video: it read on-screen text well, but its scene descriptions were unreliable. Gemini, OpenAI, and OpenRouter use the same connection as Ollama but haven't been tested with their real services.
Claude
Create a key in the Claude Console. API usage is billed separately from Claude Pro and Max plans.
ANTHROPIC_API_KEY=your-keyGemini
Create a key in Google AI Studio. Free-tier content may be used to improve Google's products, so avoid private videos on that tier.
VISION_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/
VISION_API_KEY=your-key
VISION_MODEL=gemini-3.8-flashOpenAI
VISION_BASE_URL=https://api.openai.com/v1
VISION_API_KEY=your-key
VISION_MODEL=gpt-5.4-miniOpenRouter
Choose a model that accepts images from openrouter.ai/models.
VISION_BASE_URL=https://openrouter.ai/api/v1
VISION_API_KEY=your-key
VISION_MODEL=provider/model-nameOllama (free and local)
Install Ollama, then download a vision model. Allow about 8 GB of disk space and at least 16 GB of memory, and close other heavy apps.
ollama pull gemma4:12bStart Ollama with a larger context. Its default is too small for this prompt and the images:
OLLAMA_CONTEXT_LENGTH=16384 ollama serveIf Ollama is already running in the background, quit it first. Then add this to .env:
VISION_BASE_URL=http://localhost:11434/v1
VISION_MODEL=gemma4:12b
VISION_MAX_FRAMES=12
VISION_REASONING_EFFORT=none
VISION_TIMEOUT_S=600VISION_MAX_FRAMESlimits how many images are sent per video. Fewer frames use less memory but can miss short captions.VISION_REASONING_EFFORT=noneturns off model thinking. On a 16 GB Apple Silicon Mac, one request took about 78 seconds this way and about 250 seconds with thinking, which gave somewhat better scene descriptions.VISION_TIMEOUT_Sgives slower local models enough time to answer.
Model names change often. If one is rejected, check your provider's current model list.
Usage
# A YouTube video or Short
video-to-context "https://www.youtube.com/shorts/VIDEO_ID"
# A video file on your computer
video-to-context --file path/to/video.mp4
# A file with details about where it came from
video-to-context --file ad.mp4 --sponsor "Brand" --date 2026-11-02 --caption "Post text"
# Process a video again, ignoring the saved result
video-to-context "https://youtu.be/VIDEO_ID" --forceVideos can be up to 5 minutes and 200 MB. To change these limits, edit config.py in a source install.
The command exits with 0 for a complete record, 1 for a partial record, and 2 for a rejected input.
Use it from Python
from dotenv import load_dotenv
from video_to_context.run import process
load_dotenv()
record = process(url="https://www.youtube.com/shorts/VIDEO_ID", out="store")
print(record.llm_document)
print(record.visual_context.summary if record.visual_context else "No visual analysis")Use it from AI tools (MCP)
The included MCP server lets AI tools analyse a video when you ask, for example: "Summarise this Short and tell me what its on-screen text claims."
Finish the Quick start first. The server reads your settings from ~/.video-to-context/.env and stores results in ~/.video-to-context/store.
Claude Code
claude mcp add --scope user video-to-text -- "$(command -v video-to-context-mcp)"Cursor, Claude Desktop, and other clients
Find the full command path with command -v video-to-context-mcp, then add it to the client's MCP configuration. For Cursor, that's ~/.cursor/mcp.json.
{
"mcpServers": {
"video-to-text": {
"command": "/absolute/path/to/video-to-context-mcp"
}
}
}Tools
Tool | What it does |
| Starts a background job for a YouTube link or allowed local file |
| Reports whether the job is queued, running, or finished |
| Returns the finished result as Markdown or JSON |
| Lists videos that already have results |
| Checks ffmpeg, the vision model, and file access |
Videos run one at a time in the background, because processing takes longer than most tools wait for a single call.
Security
Local files are off by default. To allow them, list specific folders in
VIDEO_TEXT_ALLOWED_DIRSin~/.video-to-context/.env.URLs are limited to YouTube.
The server runs locally over stdio and opens no network port.
Video text is untrusted. A caption could try to manipulate your AI tool, so be careful when that tool can also run commands or edit files.
Output
Each video gets its own folder, named after a hash of the file, so the same file is never processed twice:
store/<sha256>/
├── record.md Document for LLMs and people
├── record.json Full structured data for code
├── video.mp4 The video
├── frames/ Keyframes referenced by the record
└── info.json YouTube metadata (for YouTube links)record.md
Pass this file directly to a language model. It contains:
YAML metadata: source, duration, language, footage type, status, and models used.
Quality notes: anything downstream services should be careful about.
Visual analysis: a summary, the distinct settings, editing techniques, and shots.
Timeline: everything in time order, labelled by source.
Label | Meaning | Safe to quote as the video's content? |
| Transcribed speech | Yes, but it may have recognition errors |
| Captions and text in the frame | Yes |
| Text or marks that stay on screen | Yes |
| A model's description of a scene | No, it is an interpretation |
Video text is untrusted input. It's kept inside code blocks so that your prompt can treat it as data, but a caption could still contain text written to manipulate a model. Tell your downstream model not to follow instructions found in it.
record.json
The same information as structured fields, including transcript, on_screen_text, visual_context, visual_descriptions, keyframes, flags, and errors. Suspected invented speech is moved to transcript.removed_segments so you can review it.
If one stage fails, the others still finish and the record is marked partial.
Troubleshooting
What you see | What to do |
| The video may be private, removed, or age-restricted, or YouTube may be blocking the download. Download limits are stricter on cloud servers. Use |
| No vision model is set. Check that |
| The API key is wrong or expired. |
| The model didn't return usable JSON. Try a more capable model. |
Out of memory with Ollama | Lower |
| Run |
MCP tools don't appear | Use the full path from |
| Raise |
| A harmless macOS warning from two video libraries. |
| Install ffmpeg (step 1), then open a new terminal. |
It's slow | Speech-to-text runs on CPU unless you have an NVIDIA GPU. Expect a few minutes for a 2-minute video. |
Privacy and responsible use
Unless you use Ollama or no vision model, video frames are sent to the provider you choose. Check its data retention and training terms.
Videos contain people's faces and voices. The vision prompt tells the model not to identify people, and you shouldn't add face or voice identification without legal review.
YouTube's terms restrict downloading. Make sure you have the right to process a video, especially for commercial or public use.
.env,store/, and media files are ignored by git. Never commit API keys.
Development
git clone https://github.com/wani-ham/youtube-shorts-to-text-context.git
cd youtube-shorts-to-text-context
python3.12 -m venv .venv
.venv/bin/pip install -e ".[dev]"
cp .env.example .env
.venv/bin/python -m pytest
.venv/bin/python -m video_to_context.evaluateIn a source checkout, settings come from the project's .env, and the MCP server stores results in the project's store/ folder.
To score accuracy on your own test set, put each video in tests/samples/<name>.mp4 and its hand-written labels in tests/expected/<name>.json:
{"transcript": "…", "on_screen": ["…"], "no_speech": false, "rejected": null}License
Licensed under the Apache License 2.0. See NOTICE for copyright details.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Turn any video into context for AI agents: transcripts, metadata, chapters, timestamps.
Transcripts, summaries, chapters and timestamped search for video links, for AI agents.
Extract time-coded spoken transcripts, 0-3s viral hooks, visual OCR captions & UGC clone briefs.
Any video URL to LLM-ready transcript. ASR built in, no captions needed. TikTok, X, TED and more.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenancevidlizer pulls frames out of any video, image, or PDF using ffmpeg, sends them to a vision LLM, and returns a flow array — one entry per scene. Each entry tells you what happened, who was on screen, what text was visible, and what changed. If the video has audio, it transcribes it with Apple MLX Whisper and merges the speech into each step.17 PyPI4MIT
- AlicenseNot gradedqualityDmaintenanceEnables AI assistants to watch YouTube videos by extracting frames at scene changes and visual references, pairing each frame with the exact words spoken at that timestamp. Provides dense frame-transcript interleaving for any model.6MIT
- AlicenseNot gradedqualityAmaintenanceLets any LLM agent actually watch videos: a watch_video tool takes a URL or local file and returns scene-aware keyframes fused with a timestamped transcript, processed 100% locally with per-source caching.1,381 PyPI2,328MIT
- FlicenseNot gradedqualityBmaintenanceEnables AI clients to watch and hear videos from local files or YouTube, Vimeo, and Loom links by returning timestamped key frames, a transcript, a contact sheet, and a combined timeline, with all processing done locally.-