Skip to main content
Glama

vlearn

Give image-capable LLMs the ability to understand video.

An open-source, model-agnostic MCP server that turns YouTube videos into structured, timestamped, verifiable reports. Use any OpenAI-compatible endpoint, hosted or local; run fully local for $0.

Python 3.10+ License: AGPL-3.0 MCP Server OpenAI-compatible

English · 繁體中文

Quickstart · MCP Server · How it works · Configuration · FAQ


vlearn is an open-source YouTube video summarizer and MCP server for people who want to check a summary, not just read it. It combines subtitles, keyframes and a vision-language model (VLM), local or hosted, into a report where every claim carries a timestamp and a source, and where disagreements between sources are flagged instead of silently merged.

It also lets AI agents (Claude Desktop, Hermes and other MCP clients) work with video even when their model can read images but has no native video input. Instead of a raw video, the agent gets a timestamped timeline of subtitles and keyframe descriptions, and can pull any single frame on demand.

Demo

🎬 Demo GIF coming soon.

$ vlearn ingest "https://youtube.com/watch?v=qp0HIF3SfI4" --platform youtube
{'video_id': 'qp0HIF3SfI4', 'title': 'How Great Leaders Inspire Action', 'duration': 1115.0, 'subtitle_segments': 43}

$ vlearn run qp0HIF3SfI4
{'run_id': 'run_abc123', 'bundles': 13, 'conflicts': 0, 'image_frames': 9, 'cost_usd': 0.0}

$ vlearn report run_abc123
# How Great Leaders Inspire Action | Simon Sinek | TED

## Core Content
### 00:45 – 02:14
- [official_subtitle] Why is Apple so innovative? ...

## Key Moments
### 09:45 – 11:14
- [name]   Orville and Wilbur Wright
- [number] December 17th, 1903
- [number] 2.5%
- [color]  deep red curtain
- [other]  On-screen text: "Number of talks: 1"
...

(abridged; every line carries its source tag and time range)

Related MCP server: mcp-youtube

Why vlearn?

Most YouTube summarizers return a paragraph you have to trust. vlearn is built around three ideas:

  • Verifiable. Every assertion is tagged with a timestamp and a source type (official_subtitle, keyframe_ocr, keyframe_vlm, asr, danmaku). You can jump back to the evidence.

  • Honest about conflicts. When the subtitles say one thing and a keyframe shows another, vlearn arbitrates by source reliability and, if unresolved, keeps both versions.

  • It looks at the video, not just the transcript. Keyframes are chosen by scene change and described by a VLM. The image pipeline is independent of subtitles, so silent videos work too.

Typical summarizer

vlearn

Output

Prose paragraph

Structured report with assertions

Traceability

Usually none

Timestamp + source ref on every claim

Source disagreement

Not surfaced

Detected, arbitrated, both versions kept

Visual content

Often transcript only

Keyframes described by a VLM

Where data goes

Usually a cloud service

Your machine (SQLite + files)

Cost

API fees or subscription

$0 with local models

Agent integration

Rarely

15 MCP tools

Corrections

Manual editing

Persistent correction loop (vlearn review)

Quickstart

Prerequisites: Python 3.10+, FFmpeg, Deno (needed by yt-dlp). An LLM endpoint (local server or hosted API) is optional but recommended. Frame analysis needs a vision-capable model; subtitle summaries work with text-only models.

# 1. Install
git clone https://github.com/MRSogzz/vlearn
cd vlearn
pip install -e ".[media]"

# 2. (Recommended) start a local LLM; any OpenAI-compatible server works
#    Without one, vlearn falls back to regex extraction (lower quality, still free)
llama-server -m models/gemma-4-e4b.gguf --port 8080

# 3. Analyze a video
python -m app ingest "https://youtube.com/watch?v=qp0HIF3SfI4" --platform youtube
python -m app run qp0HIF3SfI4
python -m app report <run_id>

Expect roughly 2–5 minutes for an 18-minute video, mostly LLM inference. No GPU is required, but one makes summaries much better.

MCP Server

vlearn exposes its pipeline through the Model Context Protocol, so an agent can ingest a video, read the report, inspect conflicts and request a single frame at any timestamp.

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "vlearn": {
      "command": "/path/to/venv/bin/python",
      "args": ["-m", "app.mcp.server"],
      "env": {
        "VLEARN_CONFIG": "/path/to/vlearn/config/default.yaml",
        "PYTHONPATH": "/path/to/vlearn"
      }
    }
  }
}

Add to ~/.hermes/config.yaml:

mcp_servers:
  vlearn:
    command: "/path/to/venv/bin/python"        # Windows: C:/path/to/venv/Scripts/python.exe
    args: ["-m", "app.mcp.server"]
    env:
      VLEARN_CONFIG: "/path/to/vlearn/config/default.yaml"
      PYTHONPATH: "/path/to/vlearn"
    enabled: true
    timeout: 600
    connect_timeout: 60

Then ask your agent: "Analyze this YouTube video and list any conflicts: ".

Tool

Purpose

ingest_video(url, platform)

Fetch metadata and subtitles

analyze_video(video_id)

Start async analysis

get_job_status(job_id)

Poll job progress

get_report(run_id)

Read the Markdown report

get_conflicts(run_id)

List detected conflicts

read_frame_at(video_id, ts)

Extract a single frame on demand

submit_correction(...)

Submit a manual correction

list_videos(limit) / get_video(video_id)

Browse ingested videos

get_cost_summary(daily)

Cost ledger

import_trend_csv(path) / list_trend_top(n)

Trending-video workflow

preview_keyframe_selection(...)

Debug frame-level selection

preview_keyframe_selection_seek(...)

Debug seek-margin selection

arbitrate_evidence(...)

Manual arbitration

Resources: vlearn://videos/{video_id} (metadata JSON), vlearn://reports/{run_id} (full report) Prompts: analyze_youtube(url), trend_digest()

How it works

YouTube URL
    │
    ▼
INGEST        metadata + subtitles (3-tier fallback: timedtext → yt-dlp → empty)
    │
    ▼
ANALYZE       ┌─ Subtitle pipeline:  triage → window split → LLM summary
(orchestrator)├─ Image pipeline:     probe → classify → scan → VLM description   (independent of subtitles)
              └─ Fusion:             evidence bundles → consistency check
                                     → arbitration → backtrace → final summary
    │
    ▼
Structured Markdown report + SQLite

Design decisions that matter:

  • Keyframes are fetched on demand, not by downloading the whole video. Remote videos use a seek-margin strategy; frame-exact selection is kept for locally downloaded files (download_media: true).

  • Right-boundary truncation. A keyframe window never reaches into the next scene.

  • Evidence bundles use bidirectional overlap, so a short window is not swallowed by a long one.

  • Reliability-weighted arbitration: official_subtitle > keyframe_ocr > keyframe_vlm > asr > danmaku. Unresolved conflicts keep both versions.

  • Semantic color normalization. red curtain and 深紅色帷幕 count as consistent, but the report preserves the original wording.

  • Local state. Everything lives in SQLite plus files under out/. Inference runs wherever you point it, including fully local.

Configuration

Swap models by editing config/models.yaml; no code changes. Any OpenAI-compatible endpoint works. The keyframe_desc stage needs a vision-capable model; the other stages work with text-only models.

providers:
  local:
    base_url: http://127.0.0.1:8080/v1
    api_key: sk-local
    models:
      gemma:
        role: cheap
        supports_vision: true
        input_usd: 0.0
        output_usd: 0.0

routing:
  triage:        {provider: local, model: gemma}
  local_summary: {provider: local, model: gemma}
  keyframe_desc: {provider: local, model: gemma}
providers:
  local-qwen:
    base_url: http://127.0.0.1:11434/v1
    api_key: sk-local
    models:
      qwen3-27b:
        role: cheap
        supports_vision: false
        input_usd: 0.0
        output_usd: 0.0

routing:
  triage:        {provider: local-qwen, model: qwen3-27b}
  local_summary: {provider: local-qwen, model: qwen3-27b}
  keyframe_desc: {provider: local,      model: gemma}   # vision stays on Gemma
platform:
  primary: youtube
  youtube:
    download_media: false    # true = full download
  bilibili:
    enabled: false

keyframes:
  scene_threshold: 0.35      # FFmpeg scene-detection threshold
  max_window_sec: 8.0        # max window for the last change point
  min_gap_sec: 1.0           # merge dense change points
  fixed_interval_sec: 90     # fallback sampling interval
  max_frames_per_10min: 30   # frame budget

vision:
  policy: on_demand          # always | on_demand | none
  max_images_per_video: 25

cost:
  daily_usd: 2.0
  per_video_usd: 0.10
  on_exceed: degrade         # degrade | stop

publish:
  comment_enabled: false

CLI

vlearn ingest <url> --platform youtube          # metadata + subtitles
vlearn run <video_id> --profile standard        # full pipeline
vlearn report <run_id> --format md              # read the report
vlearn review <run_id> --ref <assertion_ref> --old "old" --new "new" --note "why"
vlearn cost --daily                             # cost ledger
vlearn videos --limit 20
vlearn trend import <csv> && vlearn trend run --top 20

Validation status

vlearn has been spot-checked on three videos of different types. We sampled 30 assertions from each video (90 in total) and checked every one against its source. All checks used a local 4B model, and other models have not been measured. The sample is small, so read this as a smoke test rather than a benchmark.

Metric

Result

Sampled assertions

90 (30 per video), each checked individually

Traceability (timestamp + source ref resolves to evidence)

100% (90/90)

Wrong assertions in raw VLM output

3/90 (3.3%; 95% confidence interval roughly 1% to 9%)

Wrong assertions that reached the final summary

None observed (95% upper bound about 4%)

Assertions drawn from the chart and case-study images in the sample

All checked correct (only four chart in the sample)

About the 3 wrong assertions: each was a case where the VLM described the same frame in two conflicting ways. For example, one description said the speaker held a green microphone and another said a black microphone, while the frame actually shows a black microphone with a green label. Conflict detection flagged the disagreement, the pipeline excluded those assertions, and none of them was used in the summary.

Video types covered

Type

Length

Characteristics

Humorous conference talk

about 6 min

Single-shot speaker, large projected screen, little on-screen text

Formal business talk

about 18 min

Single-shot speaker, case-study images, one chart

Music video

about 3.5 min

Multi-scene editing, lyric subtitles, stage and dance footage

Together these span very dense to very sparse subtitles, nearly static to heavily edited footage, a 5x range in length, two talk styles, and repeated-text subtitles.

Not yet tested

Content type

Why it matters

Possible risk (unverified)

Tutorials, slide decks, code demos, screen recordings

Where VLM hallucination is likely worst

A small VLM may invent chart numbers or code. Only one chart was tested, so evidence is thin

Videos without subtitles (silent demos, dance)

Exercises the subtitle = 0 fallback

May degrade to keyframe-only assertions

Short videos under 2 minutes

Tests the triage.min_duration_sec: 120 boundary

May be judged "not worth reading" and get only a light summary

Non-English videos

Subtitle language coverage

CJK-weighted assertion filtering is only tuned for Chinese and English

Interviews and podcasts (fixed camera, faces)

Redundant frames

May spend frame budget on frames with little information

News and documentaries (mixed footage)

Subtitles and visuals are tightly linked

May trigger more conflicts

Livestream recordings (over 1 hour)

Very long inputs

Many windows, so LLM call cost may grow quickly

Help wanted: if you try vlearn on one of these types, please open an issue with the video type, what worked and what did not. Each report directly improves the roadmap.

Known limitations

  • Small local models (e.g. a 4B VLM) vary run to run: on the same video, assertion counts can range from about 16 to 48. A larger model or temperature=0 helps. Differences between different videos are normal and unrelated.

  • Conflict detection catches inconsistent errors. A mistake the VLM makes the same way every time would not be flagged.

  • VLMs can over-interpret frames (for example "dark red" for "red"). Use vlearn review to correct it; corrections persist across re-runs.

  • apply_corrections does plain substring replacement, and the assertions field does not reflect corrections.

  • The video-type classifier may misjudge some talks (e.g. TED) as slide presentations.

  • CLI/MCP only; there is no web UI yet. Live streams are not supported.

FAQ

No. Without a local LLM, vlearn falls back to regex-based assertion extraction (lower quality, zero cost). Any OpenAI-compatible runtime works: llama.cpp, Ollama, LM Studio, vLLM.

Yes, edit config/models.yaml. Any OpenAI-compatible endpoint works, for example a hosted model such as GPT-4o or DeepSeek, or a local Qwen3 27B (text-only, so pair it with a vision model for frames). Claude and other APIs that are not OpenAI-format can be used through a compatible gateway such as OpenRouter or LiteLLM. Only a local 4B model has been validated so far (see Validation status).

Mostly. After a video has been ingested, vlearn run works fully offline. Only the first ingest needs the network, and not even that if subtitles are cached in cache/subs/<video_id>/.

Subtitles: any language YouTube provides. Assertion extraction works best in English and Chinese; other languages fall back to regex. Output language depends on the model you route to.

Two reasons: small VLMs over-interpret, and conflict detection normalizes deep red curtain, red curtain and 深紅色帷幕 to the same value while keeping the original text in the report. Fix wording with vlearn review.

Create an adapter in app/adapters/ and register it in AppContext.from_config() (app/context.py). The pipeline is platform-agnostic.

from app.adapters.base import PlatformAdapter, VideoMeta

class VimeoAdapter(PlatformAdapter):
    name = "vimeo"
    capabilities = {"metadata", "subtitles"}

    def get_metadata(self, url: str) -> VideoMeta: ...
    def get_subtitles(self, meta: VideoMeta) -> list[dict]: ...
out/
├── vlearn.db          # SQLite (all structured data)
├── run_xxx.md         # Markdown reports
└── frames/<video_id>/ # Extracted keyframes

Delete out/ to start fresh. vlearn.db can be backed up on its own.

When the source-reliability ranking can settle a disagreement, vlearn arbitrates and uses the more reliable source. When it cannot, vlearn keeps both versions, marks the conflict unresolved, and does not state either one as fact in the summary; you can inspect them with get_conflicts(run_id). The cost is that a report can contain fewer assertions than the raw model output. That trade is deliberate: a missing assertion is easier to live with than a wrong one that nobody can see.

No. vlearn is a local-first tool for technical users, which keeps videos on your machine and costs at zero. A multi-user web service would need auth, queues, storage isolation and cost attribution, which is a different product. The AGPL license lets you build one.

Development

pytest -q
app/
├── adapters/    platform adapters
├── keyframes/   keyframe selection (selector.py: right-boundary truncation)
├── summarize/   local LLM summarization (5-level JSON rescue)
├── evidence/    bundling, consistency, arbitration
├── fusion/      interleaved context + final summary
├── cost/        ledger, budget, router
├── storage/     SQLite + cache
├── mcp/         MCP server
└── orchestrator.py

Roadmap

  • M0 Subtitle extraction, keyframes, local summaries, Markdown report

  • M1 Triage, right-boundary truncation, evidence bundling, cost ledger

  • M2 Conflict detection, arbitration, image pipeline, manual corrections

  • M3 OCR (PaddleOCR), Bilibili adapter, trending-CSV workflow

  • M4 FastAPI layer + web UI

  • Docker image

Contributing

PRs are welcome; please open an issue first for major changes. Every new behavior needs a test.

git checkout -b feature/my-feature
pytest -q
git commit -m "feat: my feature"

Disclaimer

vlearn uses yt-dlp to fetch subtitles and frames; it does not use YouTube's official API. It is intended for personal learning and research. You are responsible for complying with the terms of service of the platforms you use and with copyright law in your jurisdiction. The software is provided as-is, without warranty.

License

AGPL-3.0. Network services built on vlearn must publish their source. For commercial licensing: <MRSogzz@gmail.com>

Acknowledgments

yt-dlp · llama.cpp · MCP Python SDK · pydantic


If vlearn saves you time, a ⭐ helps others find it.

Available Tools

15 tools
analyze_videoC

啟動異步分析,返回 job_id。

Inputs: video_id (required), profile.

arbitrate_evidenceC

對給定證據執行仲裁。summaries: [{source_type, confidence, quality_flags}]

Inputs: video_id (required), bundle_id (required), summaries (required), time_range (required).

get_conflictsC

衝突列表;unresolved 保留雙版本。

Inputs: run_id (required), only_unresolved. Returns: result

get_cost_summaryC

成本與預算餘量。

Inputs: daily.

get_job_statusD

輪詢 job。

Inputs: job_id (required).

get_reportC

讀取 Markdown 報告。

Inputs: run_id (required).

get_videoC

視頻元數據 + 源片段統計。

Inputs: video_id (required).

import_trend_csvC

導入短影音熱度 CSV。

Inputs: path.

ingest_videoC

攝取元數據 + 字幕。

Inputs: url (required), platform.

list_trend_topC

按熱度返回前 n 條。

Inputs: n. Returns: result

list_videosC

列出已攝取視頻。

Inputs: limit. Returns: result

preview_keyframe_selectionC

預覽右邊界截斷結果(幀級精確,本地完整視頻場景)。

Inputs: duration (required), scene_points (required), fps, min_gap_sec, +1 more. Returns: result

preview_keyframe_selection_seekC

預覽 seek 版的 margin 截斷結果(遠程 seek 場景)。

Inputs: duration (required), scene_points (required), min_gap_sec, max_window_sec, +2 more. Returns: result

read_frame_atA

按需讀單幀,返回 [本地 VLM 描述 + 原始圖片] 雙重視角。

include_image 參數(支持字符串形式 "true"/"false"):

  • True(默認):返回 [TextContent, ImageContent]

  • False:只返回 [TextContent]——適合客戶端處理圖片有 bug 的場景

無論 include_image 是 True 還是 False,TextContent 裡始終包含 Qwen3-VL 27B 生成的完整畫面描述 + 提取的斷言。

  • 只下載 timestamp ±2 秒的片段(~200KB),不整片下載

  • 命中 cache/frames/{video_id}/{rounded_ts}_{quality}.jpg 則直接返回

  • quality 可選:240p | 360p | 720p(默認)

Inputs: video_id (required), timestamp (required), quality, include_image.

submit_correctionD

提交人工修正。

Inputs: run_id (required), new_value (required), old_value (required), assertion_ref (required), +1 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    C2.7/5.0

    Scored across 15 tools

    Disambiguation4/5

    Most tools map to a distinct resource plus action (ingest, analyze, poll job, report, conflicts, cost, correction). The only real overlap is preview_keyframe_selection vs. preview_keyframe_selection_seek, which are near-duplicates split by local vs. remote scenario; the descriptions differentiate them but the boundary requires reading carefully.

    Naming Consistency4/5

    Names are overwhelmingly verb-first snake_case (get_report, list_videos, ingest_video, analyze_video, submit_correction, read_frame_at), giving a predictable pattern. Minor deviations: list_trend_top inverts the noun/qualifier order, and import_trend_csv embeds the object, but nothing is genuinely chaotic.

    Tool Count5/5

    15 tools sit at the top of the well-scoped range and each covers a step in the ingest → analyze → poll → report → correct pipeline, plus trend and cost views. No tool looks redundant except the deliberate preview pair.

    Completeness3/5

    The core lifecycle (ingest, async analyze, job polling, report, conflicts, correction) is covered, so no dead end in the main flow. However there is no way to delete or re-ingest a video, list/cancel running jobs, or enumerate available reports, and trend data can only be imported and read as top-n with no query or update path.

    Related MCP Connectors

    Related MCP Servers