VideoNote-Mcp packages the entire pipeline of "video link → multi-format notes" into an MCP Server + Claude Code Skill: give the agent a link, and it automatically completes download → speech transcription → visual understanding → danmaku/comments → AI summary, returning a portable note with screenshots that can be moved as a whole.

Repository: HuangYincan/VideoNote-MCP.

This project can be used end-to-end (one link → one note) or decoupled: each stage of the pipeline (download / transcription / frame extraction / comments / summary / export / enhancement / cleanup) is an independent MCP tool, so you can use just one step or simply understand the video content. No backend service needs to be started.

⚡ Quick Start

# 1) 一条命令装好 Skill + MCP（插件 marketplace，uvx 自动更新）
claude plugin marketplace add HuangYincan/VideoNote-MCP
claude plugin install videonote@videonote

# 2) 安装时 Claude Code 会逐项提示默认值（风格/转写引擎/视频理解/评论等）；
#    装完在会话里跑配置向导收尾：
/videonote-setup

# 3) （可选）LLM-Key/B 站扫码/CLI向导
# ! videonote setup

# 4) 重启会话，对 agent 说「帮我给这个视频做笔记」+ 链接

[!TIP] See docs/04-使用手册.md for four installation methods, configuration details, updates, and security.

Related MCP server: tldw-mcp

📚 Documentation

Complete instructions for installation / configuration / usage / environment variables / updates / security are archived in docs/ (the README only keeps an overview):

• 📇 Documentation Index

• 🏗️ Architecture Design

• 📖 User Manual — Installation (4 methods) · Configuration (setup wizard + CLI) · Environment variables · Updates · Security

• 📜 Changelog

🎬 Real Cases

Two end-to-end real cases: one uses AGENT direct generation and outputs a LaTeX mathnote PDF, the other uses fully automatic LLM generation to produce portable Markdown.

Case 1 · agent_direct + LaTeX mathnote (DeepSeek-V4 video)

Source: [Shanke] In-depth interpretation of DeepSeek V1~V4! Understandable for everyone.

One video + four types of external materials (paper / technical report / official WeChat announcement / open-source collection) → AGENT direct generation of a refined note, and output a LaTeX mathnote PDF (Chinese Kai font template):

Highlights: agent_direct full pipeline (no LLM key, Agent reads transcription + frame images + comments to write the note itself) · multi-source cross-integration (video × paper × technical report × open-source list) · refinement preserves the original (note.md / note_original.md dual copies) · LaTeX mathnote PDF (auto-fixes missing fonts / line-break overflow / duplicate references). Full process record: examples/agent-direct-deepseek-v4-mathnote/README.md.

Case 2 · Fully automatic LLM generation + portable Markdown (parallel multi-video)

Minimal prompt (3 Bilibili links + output directory, no parameters specified) → fully automatic runs environment check → link recognition → provider/model discovery → parameter confirmation → parallel multi-video → post-generation refinement based on subtitles, producing 3 refined portable notes (note.md + Assets/ screenshots + "Audience Perspectives" section, while keeping note_original.md for comparison).

• IELTS: breaking misconceptions + breakdown of listening/reading/writing/speaking + 179 high-frequency test words + 15 logical framework sentences

• Forensic: a 43-year forensic expert "frame-by-frame" comparison of film vs. reality, refined and expanded to 12 sections

• Transformer: detailed explanation of the self-attention mechanism, 18 screenshots distributed along the lecture timeline

Full process record: examples/note-generation-example/README.md.

🗺️ Pipeline Map

flowchart LR
    A["视频链接"] --> B["下载音视频<br/>+ 平台字幕"]
    B --> C["语音转写<br/>或直接用平台字幕"]
    B -. 可选 .-> D["逐帧画面理解<br/>关键帧 → 网格图"]
    B -. 可选 .-> E["弹幕 + 评论区"]
    C --> F["素材包<br/>转写 · 帧 · 评论"]
    D -.-> F
    E -.-> F
    F --> G["AI 总结 → Markdown 底稿<br/>正文 + 截图 + 「观众观点」"]
    G --> O1["便携笔记<br/>note.md + Assets/"]
    G --> O2["字幕导出<br/>SRT · VTT · JSON"]
    G -. Agent 生成 .-> O3["创意格式<br/>思维导图 · 闪卡 · LaTeX · typst"]
    G -. 可选 .-> O4["基于完整字幕精修<br/>保留原版对比"]

0 🔄 End-to-End Full Pipeline

End-to-end mode only needs a link: generate_note asynchronously runs the entire pipeline and returns a task_id; use the lightweight get_task_status snapshot to poll until SUCCESS/FAILED/CANCELLED (max 3 in-progress tasks per process, do not submit in parallel in the same message). cancel_note provides cooperative cancellation. "AGENT direct generation" uses prepare_note_material — it only prepares the material package, does not call the configured LLM, and the agent reads the transcription, looks at the images, and writes the note itself.

1 📥 Download & Platform Parsing

inspect_video identifies the platform (bilibili / youtube / douyin / tiktok / kuaishou / local; for anything beyond the 6 built-in platforms it returns platform:"generic" and automatically uses yt-dlp generic extraction covering 1800+ sites) + checks link validity (gives the reason if invalid) + splits Bilibili multi-part videos / YouTube playlists into per-episode URLs that can be submitted independently (no download). Platform cookies go through ! videonote login bilibili / ! videonote setup, do not pass them via MCP. Platform subtitles (including Bilibili AI subtitles) are used internally by generate_note first; there is no standalone tool.

2 🎙 Speech Transcription (ASR)

Speech transcription (ASR) is done internally by generate_note: it prefers platform subtitles (including Bilibili AI subtitles), and transcribes if no subtitles are available. Engine options: fast-whisper (local) / groq / bcut / kuaishou (cloud) / mlx-whisper (macOS Apple Silicon GPU) / funasr (best for Chinese, VAD + automatic punctuation). Engine and model management go through the CLI: ! videonote transcriber set/download; check status with get_config().

3 🖼️ Video Frame Understanding (Frame Extraction)

generate_note directly supports video understanding parameters: video_understanding=True + video_interval (default 6s) + grid_size (default [3,3]), sending the grid image to a multimodal LLM to "see" the visuals.

4 💬 Danmaku & Comments

Adding include_comments=True + comments_limit (default 20) to generate_note will organize danmaku spam and high-frequency comment section opinions into the note, adding an "Audience Perspectives" section (requires Bilibili SESSDATA; fetch failure does not block the task).

5 ✍️ AI Summary & Notes

Supports 9 styles: minimal / detailed / academic / tutorial / xiaohongshu / life_journal / task_oriented / business / meeting_minutes; format=["screenshot"] produces portable notes (note.md + Assets/, with relative references that can be moved as a whole). Provider/model/transcriber configuration always goes through the CLI (! videonote providers set / ! videonote transcriber set); read-only view via get_config(). agent_direct is generated directly by the AGENT.

6 📤 Multi-format Export

Mechanical formats use export_transcript (srt / vtt / json) — deterministic rendering (timeline conversion), no LLM cost, returns a file:// path. Creative formats (mind map / flashcards / LaTeX / typst / user-defined templates) are generated by Agent based on the MD draft + SKILL templates (LaTeX includes Math Note / English Article templates: math/science note style, English manuscript/speech outline style; typst includes the zju-lab template: science/engineering notes/experiment report/paper style, with the ZJU emblem).

7 🎛️ Audio Enhancement

merge_audio merges multiple recordings / meeting segments / multiple local videos into a 16kHz mono wav before transcription. Audio preprocessing (16kHz normalization + auto-chunking for >1800s, optional noise reduction) is off by default with zero hard dependencies. diarize_media performs speaker diarization (pyannote optional heavy dependency, requires HF_TOKEN + model authorization).

8 🗂️ Task Management & Cleanup

Each task has a folder note_results/{task_id}/: raw/ (downloaded media) + gen/ (transcription/notes/frames/exports) + control files; the global task index is in the SQLite video_tasks table (with semantic titles). list_tasks enumerates all tasks (identified by semantic title), cleanup_note(dry_run=True) checks before cleaning, cleanup_note / cleanup_all clean per-task / globally (by default keeping configuration and models), and health_check checks FFmpeg / database / whisper readiness.

flowchart TB
    DATA["data/ 数据根"] --> R["note_results/ 任务目录"]
    DATA --> DB[("video_note.db<br/>SQLite 全局任务索引")]
    R --> T1["任务 A<br/>note_results/{task_id}/"]
    R --> T2["任务 B<br/>…"]
    R --> T3["任务 C<br/>…"]
    T1 --> RAW["raw/ 原始材料<br/>音视频 · 封面"]
    T1 --> GEN["gen/ 生成材料"]
    T1 --> CTRL["status.json · result.json · manifest.json"]
    GEN --> T1A["transcript.json 转写全文"]
    GEN --> T1B["note.md 成稿笔记"]
    GEN --> T1C["Assets/ 笔记内截图"]
    GEN --> T1D["frames/ 关键帧原图"]
    GEN --> T1E["srt / vtt / json 字幕导出"]
    DB -. 索引 .-> T1

🏆 Best Practices

• Study & exam prep: end-to-end + video understanding + subtitle-based follow-up refinement to thoroughly explain the course.

• Meeting minutes: merge_audio to merge segmented recordings → diarize_media for speaker diarization → meeting_minutes style.

• Lecture deep reading: after end-to-end generation, the agent refines based on the full subtitles and fills in details by section.

• Video appreciation: enable danmaku + comment integration, the note includes an "Audience Perspectives" section.

• End-to-end: use generate_note with one link (download/transcription/summary/comments all done internally); use prepare_note_material to only prepare materials.

• Real cases: full case process records in examples.

🤝 How to Contribute

• Feature branch → PR → dev (CI smoke must be green); after dev is stable, PR → main (protected branch, requires review).

• See CONTRIBUTING.md for workflow, branch naming, and pre-commit self-checks.

🙏 Acknowledgments

Thanks to the community and all contributors, to Glama for listing the MCP server, and to all open-source dependencies and upstream pipeline projects for their inspiration.