Skip to main content
Glama

Generate Text Contexts from Youtube Shorts

Turn a short-form Youtube video into structured, timestamped context in a text format. It includes both audio and visual context.

Most language models can't watch a video. This tool gives them a faithful text version of one: what was said, what was written on screen, and what was shown, with every item labelled by source and time.

[ON-SCREEN:LOGO 00:00–00:16] EXAMPLE CHANNEL
[AUDIO 00:07–00:11] Today we're testing three ways to brew coffee at home.
[ON-SCREEN 00:08–00:10] 3 WAYS TO BREW COFFEE
[VISUAL 00:07–00:16] A person in an apron pours hot water into a glass coffee maker in a bright kitchen.

Use the output as the input to summarisation, search, fact-checking, content moderation, RAG, or your own agents.

Contents

Related MCP server: youtube-mcp

How it works

YouTube link or video file
        │
   download and check
        │
        ├── audio ──► speech-to-text (Whisper, runs locally)
        │
        └── frames ─► vision model: on-screen text, scenes, settings, editing
        │
   record.json + record.md

Quick start

1. Install the requirements

You need uv, ffmpeg, and about 5 GB of free disk space. uv installs the right Python version for you.

System

Install command

macOS

brew install uv ffmpeg

Ubuntu or Debian

sudo apt install ffmpeg && curl -LsSf https://astral.sh/uv/install.sh | sh

2. Install video-to-context

uv tool install --python 3.12 video-to-context

Add uv's tools to your PATH, then open a new terminal:

uv tool update-shell

You can also use pip install video-to-context inside a Python 3.12 environment.

3. Choose a vision model (optional)

Download the settings template:

mkdir -p ~/.video-to-context
curl -o ~/.video-to-context/.env https://raw.githubusercontent.com/wani-ham/youtube-shorts-to-text-context/main/.env.example

Open ~/.video-to-context/.env and uncomment one option from the next section. Without one, the tool still produces the transcript and reads on-screen text locally, but it won't include visual analysis.

4. Run it

video-to-context "https://www.youtube.com/shorts/VIDEO_ID"

The first run is slower because it downloads the speech and OCR models. When it finishes, it prints the path to your result in a store/ folder inside your current directory.

Choosing a vision model

You don't need Claude. Any service with an OpenAI-compatible API and image support will work.

Option

Cost

Your frames go to

Best for

Claude Haiku 5.5

About 1 cent per video (estimate)

Anthropic

Tested default

Gemini

Free tier available

Google

Trying it at no cost

OpenAI

Paid

OpenAI

Existing OpenAI users

OpenRouter

Free and paid models

The selected provider

Comparing models

Ollama

Free

Nowhere, runs locally

Private text extraction, no account

None

Free

Nowhere

Transcript and text only

Status: Claude has been tested on real videos. Ollama with Gemma 4 12B has been tested on one real video: it read on-screen text well, but its scene descriptions were unreliable. Gemini, OpenAI, and OpenRouter use the same connection as Ollama but haven't been tested with their real services.

Claude

Create a key in the Claude Console. API usage is billed separately from Claude Pro and Max plans.

ANTHROPIC_API_KEY=your-key

Gemini

Create a key in Google AI Studio. Free-tier content may be used to improve Google's products, so avoid private videos on that tier.

VISION_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/
VISION_API_KEY=your-key
VISION_MODEL=gemini-3.8-flash

OpenAI

VISION_BASE_URL=https://api.openai.com/v1
VISION_API_KEY=your-key
VISION_MODEL=gpt-5.4-mini

OpenRouter

Choose a model that accepts images from openrouter.ai/models.

VISION_BASE_URL=https://openrouter.ai/api/v1
VISION_API_KEY=your-key
VISION_MODEL=provider/model-name

Ollama (free and local)

Install Ollama, then download a vision model. Allow about 8 GB of disk space and at least 16 GB of memory, and close other heavy apps.

ollama pull gemma4:12b

Start Ollama with a larger context. Its default is too small for this prompt and the images:

OLLAMA_CONTEXT_LENGTH=16384 ollama serve

If Ollama is already running in the background, quit it first. Then add this to .env:

VISION_BASE_URL=http://localhost:11434/v1
VISION_MODEL=gemma4:12b
VISION_MAX_FRAMES=12
VISION_REASONING_EFFORT=none
VISION_TIMEOUT_S=600
  • VISION_MAX_FRAMES limits how many images are sent per video. Fewer frames use less memory but can miss short captions.

  • VISION_REASONING_EFFORT=none turns off model thinking. On a 16 GB Apple Silicon Mac, one request took about 78 seconds this way and about 250 seconds with thinking, which gave somewhat better scene descriptions.

  • VISION_TIMEOUT_S gives slower local models enough time to answer.

Model names change often. If one is rejected, check your provider's current model list.

Usage

# A YouTube video or Short
video-to-context "https://www.youtube.com/shorts/VIDEO_ID"

# A video file on your computer
video-to-context --file path/to/video.mp4

# A file with details about where it came from
video-to-context --file ad.mp4 --sponsor "Brand" --date 2026-11-02 --caption "Post text"

# Process a video again, ignoring the saved result
video-to-context "https://youtu.be/VIDEO_ID" --force

Videos can be up to 5 minutes and 200 MB. To change these limits, edit config.py in a source install.

The command exits with 0 for a complete record, 1 for a partial record, and 2 for a rejected input.

Use it from Python

from dotenv import load_dotenv
from video_to_context.run import process

load_dotenv()
record = process(url="https://www.youtube.com/shorts/VIDEO_ID", out="store")

print(record.llm_document)
print(record.visual_context.summary if record.visual_context else "No visual analysis")

Use it from AI tools (MCP)

The included MCP server lets AI tools analyse a video when you ask, for example: "Summarise this Short and tell me what its on-screen text claims."

Finish the Quick start first. The server reads your settings from ~/.video-to-context/.env and stores results in ~/.video-to-context/store.

Claude Code

claude mcp add --scope user video-to-text -- "$(command -v video-to-context-mcp)"

Cursor, Claude Desktop, and other clients

Find the full command path with command -v video-to-context-mcp, then add it to the client's MCP configuration. For Cursor, that's ~/.cursor/mcp.json.

{
  "mcpServers": {
    "video-to-text": {
      "command": "/absolute/path/to/video-to-context-mcp"
    }
  }
}

Tools

Tool

What it does

start_video_analysis

Starts a background job for a YouTube link or allowed local file

get_analysis_status

Reports whether the job is queued, running, or finished

get_video_text

Returns the finished result as Markdown or JSON

list_analyzed_videos

Lists videos that already have results

check_setup

Checks ffmpeg, the vision model, and file access

Videos run one at a time in the background, because processing takes longer than most tools wait for a single call.

Security

  • Local files are off by default. To allow them, list specific folders in VIDEO_TEXT_ALLOWED_DIRS in ~/.video-to-context/.env.

  • URLs are limited to YouTube.

  • The server runs locally over stdio and opens no network port.

  • Video text is untrusted. A caption could try to manipulate your AI tool, so be careful when that tool can also run commands or edit files.

Output

Each video gets its own folder, named after a hash of the file, so the same file is never processed twice:

store/<sha256>/
├── record.md     Document for LLMs and people
├── record.json   Full structured data for code
├── video.mp4     The video
├── frames/       Keyframes referenced by the record
└── info.json     YouTube metadata (for YouTube links)

record.md

Pass this file directly to a language model. It contains:

  • YAML metadata: source, duration, language, footage type, status, and models used.

  • Quality notes: anything downstream services should be careful about.

  • Visual analysis: a summary, the distinct settings, editing techniques, and shots.

  • Timeline: everything in time order, labelled by source.

Label

Meaning

Safe to quote as the video's content?

AUDIO

Transcribed speech

Yes, but it may have recognition errors

ON-SCREEN

Captions and text in the frame

Yes

ON-SCREEN:HEADLINE, :WATERMARK, :LOGO, :OTHER

Text or marks that stay on screen

Yes

VISUAL

A model's description of a scene

No, it is an interpretation

Video text is untrusted input. It's kept inside code blocks so that your prompt can treat it as data, but a caption could still contain text written to manipulate a model. Tell your downstream model not to follow instructions found in it.

record.json

The same information as structured fields, including transcript, on_screen_text, visual_context, visual_descriptions, keyframes, flags, and errors. Suspected invented speech is moved to transcript.removed_segments so you can review it.

If one stage fails, the others still finish and the record is marked partial.

Troubleshooting

What you see

What to do

fetch_failed

The video may be private, removed, or age-restricted, or YouTube may be blocking the download. Download limits are stricter on cloud servers. Use --file instead.

vision_not_configured in flags

No vision model is set. Check that ~/.video-to-context/.env exists and an option is uncommented.

vision_auth_failed

The API key is wrong or expired.

vision_invalid_output

The model didn't return usable JSON. Try a more capable model.

Out of memory with Ollama

Lower VISION_MAX_FRAMES or use a smaller model.

command not found: video-to-context

Run uv tool update-shell, then open a new terminal.

MCP tools don't appear

Use the full path from command -v video-to-context-mcp, then restart the client.

vision_timeout

Raise VISION_TIMEOUT_S, or set VISION_REASONING_EFFORT=none.

objc: Class AVF... is implemented in both

A harmless macOS warning from two video libraries.

ffmpeg not found

Install ffmpeg (step 1), then open a new terminal.

It's slow

Speech-to-text runs on CPU unless you have an NVIDIA GPU. Expect a few minutes for a 2-minute video.

Privacy and responsible use

  • Unless you use Ollama or no vision model, video frames are sent to the provider you choose. Check its data retention and training terms.

  • Videos contain people's faces and voices. The vision prompt tells the model not to identify people, and you shouldn't add face or voice identification without legal review.

  • YouTube's terms restrict downloading. Make sure you have the right to process a video, especially for commercial or public use.

  • .env, store/, and media files are ignored by git. Never commit API keys.

Development

git clone https://github.com/wani-ham/youtube-shorts-to-text-context.git
cd youtube-shorts-to-text-context
python3.12 -m venv .venv
.venv/bin/pip install -e ".[dev]"
cp .env.example .env
.venv/bin/python -m pytest
.venv/bin/python -m video_to_context.evaluate

In a source checkout, settings come from the project's .env, and the MCP server stores results in the project's store/ folder.

To score accuracy on your own test set, put each video in tests/samples/<name>.mp4 and its hand-written labels in tests/expected/<name>.json:

{"transcript": "…", "on_screen": ["…"], "no_speech": false, "rejected": null}

License

Licensed under the Apache License 2.0. See NOTICE for copyright details.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      vidlizer pulls frames out of any video, image, or PDF using ffmpeg, sends them to a vision LLM, and returns a flow array — one entry per scene. Each entry tells you what happened, who was on screen, what text was visible, and what changed. If the video has audio, it transcribes it with Apple MLX Whisper and merges the speech into each step.
      17 PyPI
      4
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI assistants to watch YouTube videos by extracting frames at scene changes and visual references, pairing each frame with the exact words spoken at that timestamp. Provides dense frame-transcript interleaving for any model.
      6
      MIT