Video Learning Agent

Video Learning Agent turns "learning by watching videos" into an automated engineering pipeline.

You give it a Bilibili/YouTube video link, a playlist link, or a video list file. It first tries to fetch subtitles; if there are no subtitles, it sends the task to your rented GPU cloud host to run ASR; after transcription completes, it pulls the results back locally, generates Markdown study documents, and builds a local search index.

B站 / YouTube 链接或列表
-> 优先获取字幕
-> 无字幕时上传任务到 GPU 云主机
-> GPU 下载音频并跑 faster-whisper ASR
-> 拉回 transcript.md / segments.jsonl
-> 生成可学习、可操作的 Markdown 总结
-> 建 SQLite / Chroma 搜索索引

Who it's for

• People who want to batch-process course videos

• People who want to turn public videos into study documents

• People who want to automatically generate "knowledge points + examples + tool commands + GitHub projects + AI practice projects"

• People who want Claude, Codex, and WorkBuddy to call this pipeline via MCP

Related MCP server: Open CLAW Knowledge Distiller

Features

• Supports Bilibili and YouTube

• Supports single videos, playlists, and TSV/CSV video lists

• Prioritizes fetching subtitles; only runs ASR when subtitles are unavailable

• Supports manually renting a GPU cloud host, then handing the SSH info to the Agent for automatic processing

• Remote ASR uses faster-whisper

• Default model medium

• Default GPU mode cuda + float16

• Default batch size of 20 videos

• A 4090 can try 3 concurrent ASR workers by default

• Real-time download/ASR progress viewing

• Generates Markdown study documents after pulling results back locally

• Local SQLite search, optional Chroma semantic search

• Provides an MCP server

Installation

git clone https://github.com/hahahng/video-learning-agent.git
cd video-learning-agent
python3 -m pip install -e '.[remote,mcp,test,yaml]'

The system needs to have:

ffmpeg
yt-dlp
python >= 3.10

On macOS you can install it like this:

brew install ffmpeg yt-dlp

Simplest usage: videos with subtitles

If the video has subtitles, no GPU is needed.

video-agent ingest "https://www.bilibili.com/video/BVxxx"

The command creates a job directory:

work/jobs/<job_id>/

Check status:

video-agent status <job_id>

Generate summary:

video-agent digest <job_id> --index

Search:

video-agent search "文件系统"

What if there are no subtitles: rent a GPU to run ASR

If the video has no subtitles, or the subtitle quality is too poor, rent a cloud host with an NVIDIA GPU to run ASR.

This project does not automatically buy machines for you. You open a GPU instance yourself on a cloud platform, then fill the SSH info into the config.

Recommended configuration:

• GPU: RTX 4090 / A10 / A100 / L40S all work

• OS: Ubuntu 20.04 / 22.04

• Disk: at least 50GB, 100GB+ recommended for batch runs

• Network: must be able to reach Bilibili / YouTube / Hugging Face model download URLs

• Login method: SSH password or SSH key

1. Rent a GPU cloud host

Use any GPU cloud platform that supports SSH. For example, the AutoDL console:

https://www.autodl.com/console/instance/list

The general flow is:

1. Select a GPU instance, e.g. 4090.

2. Select the Ubuntu image.

3. Power it on.

4. Find the SSH connection info in the console:

   host
   port
   user
   password 或 private key

5. First confirm locally that you can log in:

   ssh -p <port> <user>@<host>

As long as this step lets you log in, the Agent can take over the rest: installing dependencies, uploading jobs, running ASR, and pulling back results.

More details: GPU ASR usage guide

AutoDL example

If using AutoDL:

1. Open the AutoDL instance list.

2. Create or start a GPU instance.

3. 4090 is recommended; choose Ubuntu as the system image.

4. Go to the instance details page and copy the SSH login info.

5. You'll get something like this:

   host: connect.xxx.autodl.com 或平台提供的连接地址
   port: 具体端口
   user: root
   password: 实例密码

6. Test locally first:

   ssh -p <port> root@<host>

7. Once you can log in, write host/port/user into video-agent.config.yaml, and put the password in an environment variable.

2. Configure GPU SSH

Copy the config file:

cp video-agent.config.example.yaml video-agent.config.yaml

Edit video-agent.config.yaml:

jobs_root: work/jobs
data_root: data

gpu_profiles:
  my_4090:
    host: your.gpu.ssh.host
    port: 22
    user: root
    remote_root: /root/autodl-tmp/video-learning-agent
    password_env: VIDEO_GPU_PASSWORD

Don't write the password into the config file. Put it in an environment variable:

export VIDEO_GPU_PASSWORD='你的 GPU SSH 密码'

If using an SSH key:

gpu_profiles:
  my_4090:
    host: your.gpu.ssh.host
    port: 22
    user: root
    remote_root: /root/autodl-tmp/video-learning-agent
    key_filename: /Users/you/.ssh/id_ed25519

3. Batch-run a video list

TSV format example:

index	bv	title	url	duration
1	BVxxxx	第一节	https://www.bilibili.com/video/BVxxxx	
2	BVyyyy	第二节	https://www.bilibili.com/video/BVyyyy	

Start:

video-agent ingest ./videos.tsv \
  --gpu-profile my_4090 \
  --batch-size 20 \
  --asr-workers 3

If you've already created a job but haven't started remote ASR yet:

video-agent run-remote-asr <job_id> \
  --gpu-profile my_4090 \
  --batch-size 20 \
  --asr-workers 3

4. Watch progress in real time

video-agent status <job_id> --gpu-profile my_4090 --watch

Output looks like:

进度：ASR 中
音频：51 / 73
转写：18 / 73
当前：BVxxxx
GPU：82%，显存 17.4G / 24.0G
音频占用：3.9G
数据盘剩余：46G
错误：无

5. Pull back results

video-agent pull <job_id> --gpu-profile my_4090

Local results will be in:

work/jobs/<job_id>/transcripts/

Each video includes:

transcript.md
segments.jsonl

6. Generate study documents

video-agent digest <job_id> --index

Output:

work/jobs/<job_id>/digests/
data/video_learning.sqlite

If OPENAI_API_KEY is set, it will call a large model to generate a richer summary:

export OPENAI_API_KEY='你的 OpenAI API Key'
video-agent digest <job_id> --index

Without an API key, it generates a structurally complete local draft.

Summary document template

Each video generates a Markdown document:

# 编号｜总结标题

## 0. 这节课的主线流程图
## 1. 知识点
## 2. 老师例子
## 3. 中间用了什么工具、命令、工作流，我们可以学什么
## 4. GitHub 可复现项目
## 5. 我们利用 AI 可以做什么项目
## 6. 今天可以动手做什么
## 7. 学完这节应该留下什么

Section 7 will try to cover:

落地产物
输入
步骤
命令
验收标准
效率提升

MCP usage

Start the MCP server:

video-agent-mcp

Provided tools:

• ingest_url

• job_status

• run_remote_asr

• pull_transcripts

• digest_transcripts

• search_notes

Claude, Codex, and WorkBuddy can call these tools as long as they support MCP.

Optional Chroma vector store

The default search is SQLite Chinese n-gram and works offline.

If you want semantic search:

python3 -m pip install chromadb
video-agent index --docs work/jobs/<job_id>/digests --chroma
video-agent search "能做什么 AI 项目" --chroma

Security notes

• Don't write the SSH password into video-agent.config.yaml.

• Don't commit .env, video-agent.config.yaml, or work/jobs/.

• The GPU cloud host is only responsible for downloading audio and running ASR; summarization runs locally by default.

• This project does not automatically purchase, stop, or destroy cloud hosts.

Development and testing

python3 -m compileall video_learning_agent tests
python3 -m pytest -q