vlearn
Summary: This vlearn MCP server lets an agent ingest, analyze, inspect, correct, and audit YouTube video reports with timestamped evidence and on-demand keyframes.
Ingest video metadata and subtitles from a URL/platform.
Start asynchronous video analysis and poll job status.
Retrieve Markdown reports and list detected conflicts, including unresolved ones.
Read a single frame at any timestamp, returning a VLM description/assertions and optionally the image.
Submit manual corrections to report assertions.
Browse ingested videos and inspect video metadata/source statistics.
Check cost and budget usage.
Import trending short-video CSV data and list top items by heat.
Preview keyframe selection for local frame-exact and remote seek-margin scenarios.
Manually arbitrate evidence for a specific bundle and time range.
Enables ingesting and analyzing YouTube videos by fetching metadata and subtitles, generating structured timestamped reports, detecting source conflicts, and extracting frames on demand for agent inspection.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@vlearn Analyze this YouTube video and give me a timestamped report: https://youtu.be/qp0HIF3SfI4"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
vlearn
Give image-capable LLMs the ability to understand video.
An open-source, model-agnostic MCP server that turns YouTube videos into structured, timestamped, verifiable reports. Use any OpenAI-compatible endpoint, hosted or local; run fully local for $0.
English · 繁體中文
Quickstart · MCP Server · How it works · Configuration · FAQ
vlearn is an open-source YouTube video summarizer and MCP server for people who want to check a summary, not just read it. It combines subtitles, keyframes and a vision-language model (VLM), local or hosted, into a report where every claim carries a timestamp and a source, and where disagreements between sources are flagged instead of silently merged.
It also lets AI agents (Claude Desktop, Hermes and other MCP clients) work with video even when their model can read images but has no native video input. Instead of a raw video, the agent gets a timestamped timeline of subtitles and keyframe descriptions, and can pull any single frame on demand.
Demo
🎬 Demo GIF coming soon.
$ vlearn ingest "https://youtube.com/watch?v=qp0HIF3SfI4" --platform youtube
{'video_id': 'qp0HIF3SfI4', 'title': 'How Great Leaders Inspire Action', 'duration': 1115.0, 'subtitle_segments': 43}
$ vlearn run qp0HIF3SfI4
{'run_id': 'run_abc123', 'bundles': 13, 'conflicts': 0, 'image_frames': 9, 'cost_usd': 0.0}
$ vlearn report run_abc123
# How Great Leaders Inspire Action | Simon Sinek | TED
## Core Content
### 00:45 – 02:14
- [official_subtitle] Why is Apple so innovative? ...
## Key Moments
### 09:45 – 11:14
- [name] Orville and Wilbur Wright
- [number] December 17th, 1903
- [number] 2.5%
- [color] deep red curtain
- [other] On-screen text: "Number of talks: 1"
...(abridged; every line carries its source tag and time range)
Related MCP server: mcp-youtube
Why vlearn?
Most YouTube summarizers return a paragraph you have to trust. vlearn is built around three ideas:
Verifiable. Every assertion is tagged with a timestamp and a source type (
official_subtitle,keyframe_ocr,keyframe_vlm,asr,danmaku). You can jump back to the evidence.Honest about conflicts. When the subtitles say one thing and a keyframe shows another, vlearn arbitrates by source reliability and, if unresolved, keeps both versions.
It looks at the video, not just the transcript. Keyframes are chosen by scene change and described by a VLM. The image pipeline is independent of subtitles, so silent videos work too.
Typical summarizer | vlearn | |
Output | Prose paragraph | Structured report with assertions |
Traceability | Usually none | Timestamp + source ref on every claim |
Source disagreement | Not surfaced | Detected, arbitrated, both versions kept |
Visual content | Often transcript only | Keyframes described by a VLM |
Where data goes | Usually a cloud service | Your machine (SQLite + files) |
Cost | API fees or subscription | $0 with local models |
Agent integration | Rarely | 15 MCP tools |
Corrections | Manual editing | Persistent correction loop ( |
Quickstart
Prerequisites: Python 3.10+, FFmpeg, Deno (needed by yt-dlp). An LLM endpoint (local server or hosted API) is optional but recommended. Frame analysis needs a vision-capable model; subtitle summaries work with text-only models.
# 1. Install
git clone https://github.com/MRSogzz/vlearn
cd vlearn
pip install -e ".[media]"
# 2. (Recommended) start a local LLM; any OpenAI-compatible server works
# Without one, vlearn falls back to regex extraction (lower quality, still free)
llama-server -m models/gemma-4-e4b.gguf --port 8080
# 3. Analyze a video
python -m app ingest "https://youtube.com/watch?v=qp0HIF3SfI4" --platform youtube
python -m app run qp0HIF3SfI4
python -m app report <run_id>Expect roughly 2–5 minutes for an 18-minute video, mostly LLM inference. No GPU is required, but one makes summaries much better.
MCP Server
vlearn exposes its pipeline through the Model Context Protocol, so an agent can ingest a video, read the report, inspect conflicts and request a single frame at any timestamp.
Claude Desktop (claude_desktop_config.json):
{
"mcpServers": {
"vlearn": {
"command": "/path/to/venv/bin/python",
"args": ["-m", "app.mcp.server"],
"env": {
"VLEARN_CONFIG": "/path/to/vlearn/config/default.yaml",
"PYTHONPATH": "/path/to/vlearn"
}
}
}
}Add to ~/.hermes/config.yaml:
mcp_servers:
vlearn:
command: "/path/to/venv/bin/python" # Windows: C:/path/to/venv/Scripts/python.exe
args: ["-m", "app.mcp.server"]
env:
VLEARN_CONFIG: "/path/to/vlearn/config/default.yaml"
PYTHONPATH: "/path/to/vlearn"
enabled: true
timeout: 600
connect_timeout: 60Then ask your agent: "Analyze this YouTube video and list any conflicts: ".
Tool | Purpose |
| Fetch metadata and subtitles |
| Start async analysis |
| Poll job progress |
| Read the Markdown report |
| List detected conflicts |
| Extract a single frame on demand |
| Submit a manual correction |
| Browse ingested videos |
| Cost ledger |
| Trending-video workflow |
| Debug frame-level selection |
| Debug seek-margin selection |
| Manual arbitration |
Resources: vlearn://videos/{video_id} (metadata JSON), vlearn://reports/{run_id} (full report)
Prompts: analyze_youtube(url), trend_digest()
How it works
YouTube URL
│
▼
INGEST metadata + subtitles (3-tier fallback: timedtext → yt-dlp → empty)
│
▼
ANALYZE ┌─ Subtitle pipeline: triage → window split → LLM summary
(orchestrator)├─ Image pipeline: probe → classify → scan → VLM description (independent of subtitles)
└─ Fusion: evidence bundles → consistency check
→ arbitration → backtrace → final summary
│
▼
Structured Markdown report + SQLiteDesign decisions that matter:
Keyframes are fetched on demand, not by downloading the whole video. Remote videos use a seek-margin strategy; frame-exact selection is kept for locally downloaded files (
download_media: true).Right-boundary truncation. A keyframe window never reaches into the next scene.
Evidence bundles use bidirectional overlap, so a short window is not swallowed by a long one.
Reliability-weighted arbitration:
official_subtitle > keyframe_ocr > keyframe_vlm > asr > danmaku. Unresolved conflicts keep both versions.Semantic color normalization.
red curtainand深紅色帷幕count as consistent, but the report preserves the original wording.Local state. Everything lives in SQLite plus files under
out/. Inference runs wherever you point it, including fully local.
Configuration
Swap models by editing config/models.yaml; no code changes. Any OpenAI-compatible endpoint works. The keyframe_desc stage needs a vision-capable model; the other stages work with text-only models.
providers:
local:
base_url: http://127.0.0.1:8080/v1
api_key: sk-local
models:
gemma:
role: cheap
supports_vision: true
input_usd: 0.0
output_usd: 0.0
routing:
triage: {provider: local, model: gemma}
local_summary: {provider: local, model: gemma}
keyframe_desc: {provider: local, model: gemma}providers:
local-qwen:
base_url: http://127.0.0.1:11434/v1
api_key: sk-local
models:
qwen3-27b:
role: cheap
supports_vision: false
input_usd: 0.0
output_usd: 0.0
routing:
triage: {provider: local-qwen, model: qwen3-27b}
local_summary: {provider: local-qwen, model: qwen3-27b}
keyframe_desc: {provider: local, model: gemma} # vision stays on Gemmaplatform:
primary: youtube
youtube:
download_media: false # true = full download
bilibili:
enabled: false
keyframes:
scene_threshold: 0.35 # FFmpeg scene-detection threshold
max_window_sec: 8.0 # max window for the last change point
min_gap_sec: 1.0 # merge dense change points
fixed_interval_sec: 90 # fallback sampling interval
max_frames_per_10min: 30 # frame budget
vision:
policy: on_demand # always | on_demand | none
max_images_per_video: 25
cost:
daily_usd: 2.0
per_video_usd: 0.10
on_exceed: degrade # degrade | stop
publish:
comment_enabled: falseCLI
vlearn ingest <url> --platform youtube # metadata + subtitles
vlearn run <video_id> --profile standard # full pipeline
vlearn report <run_id> --format md # read the report
vlearn review <run_id> --ref <assertion_ref> --old "old" --new "new" --note "why"
vlearn cost --daily # cost ledger
vlearn videos --limit 20
vlearn trend import <csv> && vlearn trend run --top 20Validation status
vlearn has been spot-checked on three videos of different types. We sampled 30 assertions from each video (90 in total) and checked every one against its source. All checks used a local 4B model, and other models have not been measured. The sample is small, so read this as a smoke test rather than a benchmark.
Metric | Result |
Sampled assertions | 90 (30 per video), each checked individually |
Traceability (timestamp + source ref resolves to evidence) | 100% (90/90) |
Wrong assertions in raw VLM output | 3/90 (3.3%; 95% confidence interval roughly 1% to 9%) |
Wrong assertions that reached the final summary | None observed (95% upper bound about 4%) |
Assertions drawn from the chart and case-study images in the sample | All checked correct (only four chart in the sample) |
About the 3 wrong assertions: each was a case where the VLM described the same frame in two conflicting ways. For example, one description said the speaker held a green microphone and another said a black microphone, while the frame actually shows a black microphone with a green label. Conflict detection flagged the disagreement, the pipeline excluded those assertions, and none of them was used in the summary.
Video types covered
Type | Length | Characteristics |
Humorous conference talk | about 6 min | Single-shot speaker, large projected screen, little on-screen text |
Formal business talk | about 18 min | Single-shot speaker, case-study images, one chart |
Music video | about 3.5 min | Multi-scene editing, lyric subtitles, stage and dance footage |
Together these span very dense to very sparse subtitles, nearly static to heavily edited footage, a 5x range in length, two talk styles, and repeated-text subtitles.
Not yet tested
Content type | Why it matters | Possible risk (unverified) |
Tutorials, slide decks, code demos, screen recordings | Where VLM hallucination is likely worst | A small VLM may invent chart numbers or code. Only one chart was tested, so evidence is thin |
Videos without subtitles (silent demos, dance) | Exercises the | May degrade to keyframe-only assertions |
Short videos under 2 minutes | Tests the | May be judged "not worth reading" and get only a light summary |
Non-English videos | Subtitle language coverage | CJK-weighted assertion filtering is only tuned for Chinese and English |
Interviews and podcasts (fixed camera, faces) | Redundant frames | May spend frame budget on frames with little information |
News and documentaries (mixed footage) | Subtitles and visuals are tightly linked | May trigger more conflicts |
Livestream recordings (over 1 hour) | Very long inputs | Many windows, so LLM call cost may grow quickly |
Help wanted: if you try vlearn on one of these types, please open an issue with the video type, what worked and what did not. Each report directly improves the roadmap.
Known limitations
Small local models (e.g. a 4B VLM) vary run to run: on the same video, assertion counts can range from about 16 to 48. A larger model or
temperature=0helps. Differences between different videos are normal and unrelated.Conflict detection catches inconsistent errors. A mistake the VLM makes the same way every time would not be flagged.
VLMs can over-interpret frames (for example "dark red" for "red"). Use
vlearn reviewto correct it; corrections persist across re-runs.apply_correctionsdoes plain substring replacement, and theassertionsfield does not reflect corrections.The video-type classifier may misjudge some talks (e.g. TED) as slide presentations.
CLI/MCP only; there is no web UI yet. Live streams are not supported.
FAQ
No. Without a local LLM, vlearn falls back to regex-based assertion extraction (lower quality, zero cost). Any OpenAI-compatible runtime works: llama.cpp, Ollama, LM Studio, vLLM.
Yes, edit config/models.yaml. Any OpenAI-compatible endpoint works, for example a hosted model such as GPT-4o or DeepSeek, or a local Qwen3 27B (text-only, so pair it with a vision model for frames). Claude and other APIs that are not OpenAI-format can be used through a compatible gateway such as OpenRouter or LiteLLM. Only a local 4B model has been validated so far (see Validation status).
Mostly. After a video has been ingested, vlearn run works fully offline. Only the first ingest needs the network, and not even that if subtitles are cached in cache/subs/<video_id>/.
Subtitles: any language YouTube provides. Assertion extraction works best in English and Chinese; other languages fall back to regex. Output language depends on the model you route to.
Two reasons: small VLMs over-interpret, and conflict detection normalizes deep red curtain, red curtain and 深紅色帷幕 to the same value while keeping the original text in the report. Fix wording with vlearn review.
Create an adapter in app/adapters/ and register it in AppContext.from_config() (app/context.py). The pipeline is platform-agnostic.
from app.adapters.base import PlatformAdapter, VideoMeta
class VimeoAdapter(PlatformAdapter):
name = "vimeo"
capabilities = {"metadata", "subtitles"}
def get_metadata(self, url: str) -> VideoMeta: ...
def get_subtitles(self, meta: VideoMeta) -> list[dict]: ...out/
├── vlearn.db # SQLite (all structured data)
├── run_xxx.md # Markdown reports
└── frames/<video_id>/ # Extracted keyframesDelete out/ to start fresh. vlearn.db can be backed up on its own.
When the source-reliability ranking can settle a disagreement, vlearn arbitrates and uses the more reliable source. When it cannot, vlearn keeps both versions, marks the conflict unresolved, and does not state either one as fact in the summary; you can inspect them with get_conflicts(run_id). The cost is that a report can contain fewer assertions than the raw model output. That trade is deliberate: a missing assertion is easier to live with than a wrong one that nobody can see.
No. vlearn is a local-first tool for technical users, which keeps videos on your machine and costs at zero. A multi-user web service would need auth, queues, storage isolation and cost attribution, which is a different product. The AGPL license lets you build one.
Development
pytest -qapp/
├── adapters/ platform adapters
├── keyframes/ keyframe selection (selector.py: right-boundary truncation)
├── summarize/ local LLM summarization (5-level JSON rescue)
├── evidence/ bundling, consistency, arbitration
├── fusion/ interleaved context + final summary
├── cost/ ledger, budget, router
├── storage/ SQLite + cache
├── mcp/ MCP server
└── orchestrator.pyRoadmap
M0 Subtitle extraction, keyframes, local summaries, Markdown report
M1 Triage, right-boundary truncation, evidence bundling, cost ledger
M2 Conflict detection, arbitration, image pipeline, manual corrections
M3 OCR (PaddleOCR), Bilibili adapter, trending-CSV workflow
M4 FastAPI layer + web UI
Docker image
Contributing
PRs are welcome; please open an issue first for major changes. Every new behavior needs a test.
git checkout -b feature/my-feature
pytest -q
git commit -m "feat: my feature"Disclaimer
vlearn uses yt-dlp to fetch subtitles and frames; it does not use YouTube's official API. It is intended for personal learning and research. You are responsible for complying with the terms of service of the platforms you use and with copyright law in your jurisdiction. The software is provided as-is, without warranty.
License
AGPL-3.0. Network services built on vlearn must publish their source. For commercial licensing: <MRSogzz@gmail.com>
Acknowledgments
yt-dlp · llama.cpp · MCP Python SDK · pydantic
If vlearn saves you time, a ⭐ helps others find it.
Available Tools
15 toolsanalyze_videoC
啟動異步分析,返回 job_id。
Inputs: video_id (required), profile.
arbitrate_evidenceC
對給定證據執行仲裁。summaries: [{source_type, confidence, quality_flags}]
Inputs: video_id (required), bundle_id (required), summaries (required), time_range (required).
get_conflictsC
衝突列表;unresolved 保留雙版本。
Inputs: run_id (required), only_unresolved. Returns: result
get_cost_summaryC
成本與預算餘量。
Inputs: daily.
get_job_statusD
輪詢 job。
Inputs: job_id (required).
get_reportC
讀取 Markdown 報告。
Inputs: run_id (required).
get_videoC
視頻元數據 + 源片段統計。
Inputs: video_id (required).
import_trend_csvC
導入短影音熱度 CSV。
Inputs: path.
ingest_videoC
攝取元數據 + 字幕。
Inputs: url (required), platform.
list_trend_topC
按熱度返回前 n 條。
Inputs: n. Returns: result
list_videosC
列出已攝取視頻。
Inputs: limit. Returns: result
preview_keyframe_selectionC
預覽右邊界截斷結果(幀級精確,本地完整視頻場景)。
Inputs: duration (required), scene_points (required), fps, min_gap_sec, +1 more. Returns: result
preview_keyframe_selection_seekC
預覽 seek 版的 margin 截斷結果(遠程 seek 場景)。
Inputs: duration (required), scene_points (required), min_gap_sec, max_window_sec, +2 more. Returns: result
read_frame_atA
按需讀單幀,返回 [本地 VLM 描述 + 原始圖片] 雙重視角。
include_image 參數(支持字符串形式 "true"/"false"):
True(默認):返回 [TextContent, ImageContent]
False:只返回 [TextContent]——適合客戶端處理圖片有 bug 的場景
無論 include_image 是 True 還是 False,TextContent 裡始終包含 Qwen3-VL 27B 生成的完整畫面描述 + 提取的斷言。
只下載 timestamp ±2 秒的片段(~200KB),不整片下載
命中 cache/frames/{video_id}/{rounded_ts}_{quality}.jpg 則直接返回
quality 可選:240p | 360p | 720p(默認)
Inputs: video_id (required), timestamp (required), quality, include_image.
submit_correctionD
提交人工修正。
Inputs: run_id (required), new_value (required), old_value (required), assertion_ref (required), +1 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 15 tools
Most tools map to a distinct resource plus action (ingest, analyze, poll job, report, conflicts, cost, correction). The only real overlap is preview_keyframe_selection vs. preview_keyframe_selection_seek, which are near-duplicates split by local vs. remote scenario; the descriptions differentiate them but the boundary requires reading carefully.
Names are overwhelmingly verb-first snake_case (get_report, list_videos, ingest_video, analyze_video, submit_correction, read_frame_at), giving a predictable pattern. Minor deviations: list_trend_top inverts the noun/qualifier order, and import_trend_csv embeds the object, but nothing is genuinely chaotic.
15 tools sit at the top of the well-scoped range and each covers a step in the ingest → analyze → poll → report → correct pipeline, plus trend and cost views. No tool looks redundant except the deliberate preview pair.
The core lifecycle (ingest, async analyze, job polling, report, conflicts, correction) is covered, so no dead end in the main flow. However there is no way to delete or re-ingest a video, list/cancel running jobs, or enumerate available reports, and trend data can only be imported and read as top-n with no query or update path.
Related MCP Connectors
YouTube for AI agents: transcripts, comments, video stats, channel monitor, Shorts captions.
YouTube transcripts, search, channel/playlist listings and upload tracking for AI agents.
YouTube transcripts, search, channels, playlists and bulk transcript jobs for AI agents. 14 tools.
Build, run, schedule, and publish AI video pipelines to YouTube and TikTok from any MCP client.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceEnables AI models to interact with YouTube content through 18 MCP tools for videos, transcripts, channels, and playlists.-
- AlicenseNot gradedqualityFmaintenanceGives MCP clients access to YouTube video transcripts and metadata. It lets AI agents read and summarize video content from a URL.8 npmMIT
- FlicenseNot gradedqualityAmaintenanceEnables users to ingest YouTube videos into a persistent, cross-referenced knowledge base by extracting transcripts, analyzing scene-change frames with vision models, and providing agentic search and cited answers through MCP clients like Claude and Copilot.-
- AlicenseNot gradedqualityCmaintenanceLets agents watch public YouTube videos and answer questions about their audio and visuals, using Gemini for transcript-aware or fixed-frame analysis.1MIT