MultiMediaGrep
MultiMediaGrep turns local video, audio, and documents into agent-readable material (storyboard grids, page images, time/page-aligned text) via 6 MCP tools — it never downloads anything and never calls an LLM itself.
media_probe— inspect a local file for free (no image tokens): type, duration, resolution, page count, codec, tracks, ASR/OCR availability, text-layer reliability, audio health, plus one outline (sidecar → native bookmarks/chapters) and annotation count.media_overview— the main action: storyboard grids + time-aligned text for video, timestamped transcript for audio, page thumbnails + per-page text for documents; text is previewed inline (whole / skim / sampled / outline) and the full text is written to disk (content.md,segments.jsonl,manifest.json).media_zoom— a single high-resolution frame or cropped page region (~1,900 tokens) — the only tool meant for reading text on screen.media_locate— find a phrase/keyword: timestamps for video/audio, page number + hit count + snippet for documents; also searches annotation notes; text-only by default.media_bookmark— write an outline/chapters/bookmarks: in-place PDF outline rewrite,.mmgrep.bookmark.vttfor video/audio, and a.mmgrep.bookmark.jsonsidecar as the source of truth (add/remove/update, all-or-nothing).media_annotate— write notes/comments/annotations: real in-place PDF annotation objects,.mmgrep.annotation.vttfor video/audio, plus a.mmgrep.annotation.jsonsidecar (author defaults to the MCP client name).
It reads local absolute paths only, prefers subtitles over ASR, refuses to hallucinate (silent audio is never sent to ASR, missing OCR/ASR is reported explicitly), supports stdio and Streamable HTTP transports, and the two write tools genuinely modify files — back up originals first.
Configures ElevenLabs as an ASR backend to transcribe local audio/video files, with configurable API key, base URL, model, language, and timeout.
Supports OpenAI /audio/transcriptions-compatible ASR services for transcribing local audio/video, using a configurable API key, base URL, and model.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@MultiMediaGrep probe /Users/me/Movies/demo.mp4 and show me a storyboard overview"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
MultiMediaGrep
给 agent 的本地媒体感知工具(MCP server)。
把本地已有的视频 / 录音 / 文档,变成 agent 能直接消化的材料:可直接看的图(分镜宫格、特写裁切) + 按时间/页码对齐的文本清单。server 自己不调 LLM——理解由 agent 的多模态模型完成。
这是什么 / 不是什么
是 | 不是 |
一个 MCP server(默认 stdio,可选 Streamable HTTP),只吃本地绝对路径 | 不下载、不检索、不登录、不替你取数(取数属于 IntelligenceAtlas) |
只做「材料化」:出图 + 出带时间码/页码的文本清单 | 不调 LLM:不做总结、不做问答、不做转述 |
ASR 与文档解析/OCR 走你自己配的外部服务与 key | 不内置本地 ASR / OCR 模型(mlx-whisper、rapidocr 都不再维护) |
缓存只写自己的缓存目录;写工具只在源文件旁写 | 不写别的状态:没有服务器数据库、没有版本历史;唯一例外是写工具的 |
失败也是明说的:没配 ASR 就只给画面,扫描件没配 OCR 就只给页图,并在结果里写清楚原因—— 宁可说「没有文字」,也不编一段台词。
Related MCP server: video-tools-mcp
工具(6 个,按动作命名,跨媒体复用)
工具 | 干什么 |
| 探媒体:时长/分辨率/页数/编码、可用轨道、文字层可靠不可靠、能否做 ASR/OCR、一份目录(sidecar / PDF 书签 / 章节)与批注条数。不花图片 token |
| 总览:视频出分镜宫格图,录音出带时间轴转写,文档出页面图 + 每页文本;额外返回全文落盘路径。内联文字是预览(whole / skim / sampled 三档,另有 outline 目录视图), |
| 特写:取指定时间点/区间的单帧或文档某页,可裁切放大,用于读字(约 1900 token/张,最贵) |
| 定位:给一句文本/关键词,返回它在视频/录音里的时间戳;文档返回页码 + 命中次数 + 命中附近的片段(不倾倒整页);也会搜批注 note |
| 写书签 / 目录 / 章节:PDF 原地改写 outline;视频、录音写 |
| 写批注 / 注释 / 评论:PDF 原地写真正的批注对象(可新建 |
前四个只读;后两个会真的改文件,调用前先看下一节。 用法指引(先 probe → 再 overview → 只 zoom 重要部分 → 长素材先收窄范围;写之前先定位)写在工具描述与 server instructions 里。
会写你的文件(0.3.0 起)
media_bookmark / media_annotate 是对「不写用户文件」的明确例外:只写 *.mmgrep.* sidecar / VTT,外加 PDF 文件本身的原地修改——
PDF 原地修改:outline / 批注对象直接写进源 PDF,不生成副本(页面正文不重写)。
视频、录音文件从不修改:只写同名
.mmgrep.bookmark.vtt/.mmgrep.annotation.vtt。源文件旁会多出 sidecar:
<file>.mmgrep.bookmark.json/<file>.mmgrep.annotation.json。 它们是唯一真相源;sidecar 一旦存在,文件自带的目录 / 批注在下次写入时会被 sidecar 覆盖。先保护原件:动文件前先
git commit(或另做备份)。写入失败时源文件保持原样: 临时文件 → 校验 → rename,任何一项检查失败都不会留下半成品。
0.3.0 有什么新东西(API 无破坏性变更)
新增两个写工具:
media_bookmark(目录 / 章节 / 书签)与media_annotate(批注 / 注释 / 评论)。 四个原有工具的参数与调用方式不变;没有新增环境变量,也没有服务器状态。唯一需要跟着改的是目录文本 格式:0.2.0 的- 标题 — page N统一成b1 p.1 标题/b1 00:12:34 标题(见下)。sidecar 是唯一真相源:每次写入在源文件旁写
<file>.mmgrep.bookmark.json/<file>.mmgrep.annotation.json;条目带稳定 id(b1…/n1…),首次写入自动导入文件自带的目录 / 批注。同时嵌入文件本身:PDF 原地改写 outline / 写入真正的批注对象(不出副本,正文不动); 视频、录音文件从不修改,只写同名
.mmgrep.bookmark.vtt/.mmgrep.annotation.vtt;图片只有 sidecar。读侧统一:
media_probe/media_overview/media_locate只展示一份目录,来源优先级为 sidecar → 文件自带(PDF 书签 / 章节)→ heading 生成(仅 overview);行格式统一为b3 p.12 标题/b3 00:12:34 标题,有 sidecar 时目录末行是full detail: <sidecar 路径>。 probe 报批注条数,overview 把批注挂在页 / 时间段旁,locate 也会搜批注 note。外部依赖:写 PDF(outline / 批注)与读 PDF 批注需要
mutool(mupdf-tools);读 PDF 书签仍用qpdf(可选)。完整的写入语义、检查项与升级说明见
docs/RELEASE-NOTES-v0.3.0.md。
0.2.0 有什么新东西(有破坏性变更)
文档 provider 拆成两个角色:
MMGREP_OCR_PROVIDER(只处理无文字层页,默认none)与MMGREP_TYPESET_PROVIDER(只处理有文字层页、恢复结构不重识别字符,默认builtin)。MMGREP_DOC_PROVIDER/MMGREP_DOC_API_KEY/MMGREP_DOC_BASE_URL/MMGREP_DOC_MODEL已删除,启动时会 warn 并指出替代变量。来源新增
text(原生文字层 / 纯文本,字符精确、可原样引用)与mixed(逐页来源不一致); 文档逐页按textOrigin映射来源,segments[]/matches[]里能看到textOrigin。内置目录:
media_probe展示 PDF 书签(物理页 + 打印标签)与视频/录音章节;media_overview的view参数支持auto/outline/text(视频没有章节时不生成目录)。内联预览分层:文本三档 whole → skim(文档:全部 heading + 段首句)→ sampled(等距抽样 + 写明未覆盖页),另有 outline 目录视图; 全文路径固定出现在 text content 末尾,读全文去读
content.md。坏文字层可强制重识别:probe 报
text layer: looks unreliable (...)时用ocr: "force"; provider 结果按页缓存,force 单独成键、不覆盖 auto。文档定位不再倾倒整页:返回页码 + 命中次数 + 命中附近的片段(
hits/snippet/fuzzy), 视频 / 录音仍是score。HTTP 客户端可覆盖内联上限:请求头
x-mmgrep-max-text-inline-chars(硬上限200000)。
完整的破坏性变更与迁移方法见 docs/RELEASE-NOTES-v0.2.0.md。
环境要求
依赖 | 必需性 | 缺了会怎样 |
Node.js ≥ 20(ESM,零构建步骤) | 必需 | 起不来 |
ffmpeg + ffprobe(同一个发行版里通常一起装) | 视频 / 录音 / 文档都需要 | 返回 |
poppler( | 只有读 PDF 时需要 | PDF 相关调用返回 |
mupdf-tools( | 写 PDF 的目录 / 批注时需要;读 PDF 自带的批注也用它 | 写 PDF 时返回 |
安装示例:
# macOS
brew install ffmpeg poppler qpdf mupdf-tools # 文档需要 poppler;写 PDF / 读批注需要 mupdf-tools
# Debian / Ubuntu
sudo apt install ffmpeg poppler-utils qpdf mupdf-tools
# Windows(winget)
winget install Gyan.FFmpeg # PDF 另装 poppler(无官方 winget 包时用 scoop install poppler)可选:把可执行文件放在 PATH 之外时,用 MMGREP_FFMPEG_PATH / MMGREP_FFPROBE_PATH 指定绝对路径。
读 PDF 书签(probe 的目录)另需 qpdf;写 PDF 时它只用于写入后校验(没有 qpdf 会改用 mutool
重新打开)。缺了 qpdf 只是读不到书签,正文与页图不受影响;写 PDF 的目录 / 批注与读 PDF 批注所需的
mutool 见上表。
外部服务(可选):ASR(ElevenLabs 或任何 OpenAI /audio/transcriptions 兼容服务)与文档解析(MinerU)
由你用环境变量配置。文档解析拆成 OCR / typeset 两个角色:OCR 默认 none,typeset 默认本地 builtin(不联网)——
不配也能用,只是扫描页拿不到文字(详见下面「环境变量」)。
安装与运行
# 1) 免安装(发布到 npm 之后)
npx -y multimediagrep-mcp
npx -y multimediagrep-mcp会拉起 server 并阻塞等待 stdio 上的 MCP 消息——它没有交互界面, 通常由 MCP client 自动拉起。想确认它起没起得来,就手动跑一次,看 stderr 有没有启动横幅:[mmgrep] info: multimediagrep@0.3.0 ready on stdio [mmgrep] info: cache: /Users/you/.cache/multimediagrep (max 10737418240 bytes) [mmgrep] info: asr: none (no API key) · ocr: none · typeset: builtin [mmgrep] info: backends: video=ready audio=ready document=ready [mmgrep] info: image budget: 4000 tokens/call, maxEdge 1600px
# 2) 从 checkout 直接跑
git clone https://github.com/flmaximwang/MultiMediaGrep.git && cd MultiMediaGrep
npm install
node src/index.js# 3) 从 GitHub 装(npm 还没发布时)
npx -y github:flmaximwang/MultiMediaGrep # 临时拉取并启动
npm i -g github:flmaximwang/MultiMediaGrep # 装成本地命令 multimediagrep-mcp
npm i -y https://github.com/flmaximwang/MultiMediaGrep/releases/download/v0.3.0/multimediagrep-mcp-0.3.0.tgz # 只装 release 里的运行时包从 GitHub 源码装会把整个仓库拉下来(源码与文档),首次安装比 npm 慢; release 里的
.tgz只含运行时(src + 署名材料)。
stdio 是默认传输方式;HTTP 是显式开启的选项。 不传 --http、也不设 MMGREP_TRANSPORT=http
时,server 不会监听任何端口,仍由 MCP client 用 stdio 拉起——现有配置不需要任何改动。
HTTP 模式见下一节。
HTTP 模式
不想让宿主用 stdio 拉起、或要让同一台机器上的多个进程共用一个 server 时,用 Streamable HTTP:
# 默认只监听 127.0.0.1:3927;stdio 仍是默认,必须显式 --http
npx -y multimediagrep-mcp --http
# 换个端口,并设置 Bearer 令牌(设置后每个请求都要带 Authorization)
npx -y multimediagrep-mcp --http --port 4000 --token '<你的令牌>'
# 也可以用环境变量(命令行优先于环境变量)
MMGREP_TRANSPORT=http MMGREP_HTTP_PORT=4000 MMGREP_HTTP_TOKEN='<你的令牌>' \
npx -y multimediagrep-mcp端点固定为
POST /mcp,默认地址http://127.0.0.1:3927/mcp;GET /mcp→ 405, 其它路径 → 404。配了令牌时,未授权的请求在判断方法之前就先回 401。启动横幅会写实际地址和
auth: bearer set|unset(只报有没有配,不打印令牌值)。请求必须同时接受 JSON 和 SSE:
Accept: application/json, text/event-stream;Content-Type: application/json。缺了分别回 406 / 415。令牌只从
Authorization: Bearer <令牌>读,不接受 query string(?token=会被忽略并回 401)。HTTP 模式是无状态的:每个请求独立处理,不在内存里保留会话。
最小宿主配置(Claude Code 的 claude mcp add):
claude mcp add --scope user --transport http multimediagrep \
http://127.0.0.1:3927/mcp \
--header "Authorization: Bearer <你的令牌>"通用 JSON 形状(多数宿主叫 mcpServers;可复制的一份见
examples/mcp-config-http.json):
{
"mcpServers": {
"multimediagrep": {
"type": "http",
"url": "http://127.0.0.1:3927/mcp",
"headers": { "Authorization": "Bearer <你的令牌>" }
}
}
}监听非回环地址(如
--host 0.0.0.0)时必须配令牌。没配时 server 仍会启动,只在启动横幅里 记一条 warning,端点对所有能连到该地址的人开放。不要在没有 TLS 的网络上裸跑; 反向代理与 curl 自测见docs/install.md§2.6。
接入宿主
Claude Code
# 用户级(所有项目可用)
claude mcp add --scope user multimediagrep \
-e MMGREP_ASR_PROVIDER=elevenlabs \
-e MMGREP_ELEVENLABS_API_KEY=sk-你的key \
-- npx -y multimediagrep-mcp
# 项目级(写进项目里的 .mcp.json,随仓库走)
claude mcp add --scope project multimediagrep -- npx -y multimediagrep-mcp.mcp.json 片段(claude mcp add --scope project 生成的就是这个形状):
{
"mcpServers": {
"multimediagrep": {
"command": "npx",
"args": ["-y", "multimediagrep-mcp"],
"env": {
"MMGREP_ASR_PROVIDER": "elevenlabs",
"MMGREP_ELEVENLABS_API_KEY": "sk-替换成你的key",
"MMGREP_OCR_PROVIDER": "mineru",
"MMGREP_TYPESET_PROVIDER": "builtin",
"MMGREP_MINERU_API_KEY": "替换成你的token"
}
}
}
}Hermes
Hermes 用同名的 hermes mcp add 流程(stdio server):
hermes mcp add multimediagrep \
--command npx \
--env MMGREP_ASR_PROVIDER=elevenlabs MMGREP_ELEVENLABS_API_KEY=sk-你的key \
--args -y multimediagrep-mcp它会先连一次、列出工具让你勾选,然后写进 ~/.hermes/config.yaml 的 mcp_servers:。
加完用 hermes mcp test multimediagrep 验连通,用 /reload-mcp 让已经开着的会话也加载上。
--args之后的参数原样透传给命令,所以--args必须放在最后。
其它 MCP client
任何吃标准 MCP client 配置的宿主都用同一段 JSON(键名通常是 mcpServers):
可复制的一份见 examples/mcp-config.json,
各宿主的完整写法见 docs/install.md §2。
每请求凭据(仅 HTTP 模式)
HTTP 端点常被多个调用方共用,provider 的 key 不必都写死在 server 的环境变量里: 每个请求可以在头里临时覆盖 ASR / 文档 provider 的角色、key、基址和模型。
请求头 | 作用 | 对应环境变量 |
| 本次请求用哪个 ASR provider: |
|
| ElevenLabs key |
|
| ElevenLabs 基址 |
|
| ElevenLabs 模型 |
|
| OpenAI 兼容服务的 key |
|
| OpenAI 兼容服务的基址 |
|
| OpenAI 兼容服务的模型 |
|
| 通用兜底 key(旧共用变量语义) |
|
| 本次请求的 OCR 角色 provider: |
|
| 本次请求的 typeset 角色 provider: |
|
| MinerU token(两个角色共用) |
|
| MinerU 基址(两个角色共用) |
|
| MinerU 模型(两个角色共用) |
|
| 本次请求的内联文本上限(设置头,不是凭据):正整数,硬上限 |
|
优先级(逐字段):该 provider 的专属请求头 > 该 provider 的环境变量 >
服务器启动配置(仅当 provider 相同)> provider 内置默认值。ASR 仍保留通用
x-mmgrep-asr-api-key 兜底;文档角色已删除通用 key 头。
请求头切换 provider 后只读新 provider 的变量组,不会借用另一个 provider 的 key;
头里写了未知的 provider id → HTTP 400(JSON-RPC -32602)。空头(空串 / 全空白)等于没带。
要让某个 provider 的专属头生效,得先用 x-mmgrep-asr-provider /
x-mmgrep-ocr-provider / x-mmgrep-typeset-provider 选中它(或服务器本身就选了同一个)。
只带 provider 专属 key 头、provider 仍是 none 时,
这个头不会自动启用 provider;provider 的选择只对当前请求生效。
安全:这些头的值只活在本次请求里——不写日志、不进缓存、不出现在响应里。server 会把本次请求
带过的所有凭据头值从日志里替换成 [redacted]。不要把真实 key 写进多人共享的客户端配置:
更稳妥的做法是让宿主连一个你自己控制的代理,由代理注入凭据头。
给第一次来的 agent:5 分钟上手
给一个没有上下文、只想「看看这个工具能干什么」的 agent:
装起来(宿主负责拉起它,不用自己开端口):
Claude Code:
claude mcp add --scope user multimediagrep -- npx -y github:flmaximwang/MultiMediaGrep其它 client:把
examples/mcp-config.json里的command/args换成上面那条命令即可。
确认环境:宿主里需要
ffmpeg+ffprobe在 PATH 上;读 PDF 还要poppler/mutool/gs任意一个; 写 PDF 的目录 / 批注、读 PDF 批注还要mutool(mupdf-tools)。缺了会返回TOOL_NOT_FOUND,不会静默出错。先探后看(省图片 token):
media_probe({ path })→ 类型、时长、有没有字幕轨、能不能 ASR/OCR。不花任何图片 token,先做这一步。media_overview({ path })→ 视频出分镜宫格图 + 时间对齐文本;录音出带时间轴转写;文档出页图 + 每页文字。文档的文字层可能是坏的(早先劣质 OCR 写进去的):probe 会给出
text layer: looks unreliable (...),这时用media_overview({ path, pages, ocr: "force" })/media_locate({ path, query, ocr: "force" })要求重新识别(需要已配置 OCR provider)。media_zoom({ path, at })→ 只在「画面上有字要读」时用,最贵(约 1900 token/张)。media_locate({ path, query })→ 「那句话在哪说的」,纯文本、零图片。想指定字幕来源时给
media_probe/media_overview/media_locate传subtitle:本地字幕文件绝对路径、字幕轨 index(非负整数),或语言码(如zh/en);显式指定不可用会直接报错,不会静默换成别的来源。要写目录 / 批注时才用
media_bookmark({ path, add: [...] })/media_annotate({ path, add: [...] }):PDF 原地改,视频 / 录音只写同名 VTT,所有类型都在源文件旁写*.mmgrep.*sidecar。
记住四件事:
只吃本地绝对路径,它不下载任何东西;
server 自己不调 LLM,理解由你的模型完成;
字幕优先于 ASR:字幕通常比 ASR 准,但可能是自动生成的(专名、数字要核对画面);ASR 是机器转写(人名、数字容易错,别当原话引用);
写工具会改文件:动原件前先
git commit(或备份);sidecar 一旦存在就是唯一真相源。
长素材先收窄:视频/录音用
start/end,文档用pages(如"3-10,15"),再跟着结果里的continuation往下走。
没有 ASR key 也能用:只是拿不到文字层(文件自带字幕就照样出文本)。
环境变量
全部前缀 MMGREP_,用 env | grep MMGREP_ 查看当前值。key 只写在你自己的配置文件里,
server 不会打印它(日志与错误里只出现 set / unset)。
变量 | 默认值 | 作用 |
|
| 传输方式: |
|
| HTTP 监听地址。非回环地址必须配令牌;没配时只记 warning,端点仍会接受未认证请求 |
|
| HTTP 监听端口(1–65535;非法值回落 |
| 空 | Bearer 令牌。设置后每个请求必须带 |
|
| 缓存根目录(抽帧、页图、转写、全文落盘都在这)。 |
|
| 缓存上限,支持 |
|
| ASR 后端: |
| 空 | ElevenLabs 的 API key( |
|
| ElevenLabs 的 API 基址 |
|
| ElevenLabs 转写模型名 |
| 空 | OpenAI 兼容服务的 API key( |
|
| OpenAI 兼容服务的 API 基址;自建服务填自己的地址 |
|
| OpenAI 兼容服务的转写模型名 |
|
| 强制语言(BCP-47,如 |
|
| ASR 单次请求超时 |
|
| OCR 角色:只处理没有文字层的页,以及工具参数 |
|
| typeset 角色:只处理有文字层的页,恢复结构、不重识别字符。 |
|
| 外部 typeset provider 只收双栏 / 复杂页,简单页交给本地 builtin |
| 空 | MinerU 的 token(两个角色共用;https://mineru.net/apiManage/token) |
|
| MinerU 的 API 基址(两个角色共用) |
| 空 | 解析模型名;留空 = 用服务端默认(MinerU 可选 |
|
| 文档解析单次请求超时(3 分钟)。MinerU 是「上传 → 排队 → 解析 → 下载」整段墙钟时间,排队久就调大 |
|
| 单次调用的图片 token 软预算。超了自动模式会降形,显式模式只警告 |
|
| 单张图最长边的全局上限(px)。请求超过 → |
|
| 日志级别: |
|
| 内联文本预览上限;HTTP 模式可被 |
|
| 单次调用的并行任务数 |
传输相关的命令行开关(--http / --host / --port / --token)优先于上表的环境变量。
旧的
MMGREP_ASR_API_KEY/MMGREP_ASR_BASE_URL/MMGREP_ASR_MODEL仍会在 对应的 provider 专属变量未设置时兜底,但启动时会记一条弃用 warning。 0.2.0 已删除MMGREP_DOC_PROVIDER/MMGREP_DOC_API_KEY/MMGREP_DOC_BASE_URL/MMGREP_DOC_MODEL:设置这些变量不会再生效,启动时会 warn 并指出新变量名。 同一进程里可以同时存在多家 provider 的 key,server 只读当前选中 provider 的那一组。
另有 MMGREP_FFMPEG_PATH / MMGREP_FFPROBE_PATH(默认按 PATH 查找)、
MMGREP_AUDIO_MIN_DBFS / MMGREP_AUDIO_MIN_SPEECH_RATIO(音频体检阈值)、
MMGREP_PROGRESS_NOTIFICATIONS(默认 true)等,完整清单见
docs/install.md §3.5 与 docs/mcp-contract.md §12。
第一次跑一遍
假设有一个本地视频 /abs/path/demo.mp4:
1. 探(不花图片 token,几毫秒返回)
{ "path": "/abs/path/demo.mp4" }返回里看五件事:媒体类型、时长、有没有字幕轨、capabilities.asr.available / capabilities.ocr.available,以及文件自带的目录(PDF 书签 / 章节)。
录音还会给一段 audioHealth 体检——如果结论是 too-quiet / silent,
转写会故意是空的(避免 ASR 编出一整段幻觉台词),这时该做的是换一份清楚的录音,而不是继续追问。
2. 总览(主体动作)
{ "path": "/abs/path/demo.mp4", "cells": 9, "images": 2 }出 2 张 3×3 的分镜宫格图 + 一份按格对齐的文本清单(每行末尾标了来源:subtitle / text / ocr / asr / none;整份文档来源不一致时 header 是 mixed)。
全文不内联,而是落盘成三个主文件(另加可选的 document.json)并把路径返回给你:
文件 | 内容 |
| 人读 Markdown,按时间区间分节 |
| 一行一个 segment(JSON),机器读 |
| 工具、窗口、转写来源、图片描述、警告、计数 |
读这些文件要自己分块总结,不要为了拿更多内联文本反复调工具。
3. 特写(贵,只在「这里有事」时用)
{ "path": "/abs/path/demo.mp4", "at": 72.5 }宫格一格只有几百像素,里面的字基本看不清;要读画面上的字必须 zoom,
必要时再加 crop({x, y, width, height},全是 0..1 的比例,原点左上)裁出那块区域。
4. 定位(纯文本,零图片)
{ "path": "/abs/path/demo.mp4", "query": "那句话是在哪说的" }返回它在视频 / 录音里的时间戳(毫秒 + 秒两种表示);文档返回页码、命中次数与命中附近的片段——不会把整页正文倒出来。
长素材先收窄。 超过 30 分钟的视频、超过 30 页的文档,先用
start/end(视频、录音)或pages(文档,如"3-10,15")取一小段,再跟着结果里的continuation提示一页页往下走。 一次 overview 硬上限是 4 张图、每张 16 格、单次共 64 格、单图最长边 1600px—— 一张「两小时电影的大宫格」既看不清又剧透。
5. 写(可选,会修改文件)
{ "path": "/abs/path/demo.mp4", "add": [{ "title": "Opening", "start": 0, "end": 12 }] }写入 sidecar <file>.mmgrep.bookmark.json 与 <file>.mmgrep.bookmark.vtt;之后 probe / overview / locate
的目录都来自 sidecar(末行 full detail: <sidecar 路径>)。批注同理用 media_annotate 写
<file>.mmgrep.annotation.json / .annotation.vtt。PDF 会原地改写 outline / 批注对象,视频 / 录音文件本身不动。
依赖、许可与上游致谢
运行时依赖:
@modelcontextprotocol/sdk(MIT)、zod(MIT)。没有构建步骤,纯 ESM。系统依赖:ffmpeg / ffprobe(不打包、不链接,按独立子进程调用)、poppler 或 mutool 或 gs(只读 PDF 时用); mutool(mupdf-tools)还用于原地写 PDF 的 outline / 批注、以及读取 PDF 批注;qpdf 可选,用于读 PDF 书签与写入后校验。
可选外部服务:ElevenLabs(ASR)、MinerU(文档解析)。都是用户自备 key,代码里不含它们的凭据。
许可:本项目 Apache-2.0(
LICENSE)。派生源码:
src/backend/video/下的geometry.js、frames.js、sheet.js、shots.js以及storyboard.js的mapPool派生自 Apache-2.0 的 dsh-video-storyboard v0.5.9(已修改)。 被改动的文件顶部都有Modifications:段落(Apache-2.0 §4(b))。 完整署名见NOTICE与CREDITS.md。
开发
纯 ESM JavaScript(Node ≥20),零构建步骤。规格与决策只在 beads(bd),不在本仓库写交接文档。
npm install
npm run check # 语法检查(src/、test/、scripts/、fixtures/ 全量 node --check)
npm test # node --test
npm run fixtures # 生成测试素材(ffmpeg 脚本生成,仓库不含任何第三方媒体)工具契约(参数、返回结构、错误码、图片预算)见 docs/mcp-contract.md;
安装、provider 配置、排障与验证清单见 docs/install.md。
Available Tools
6 toolsmedia_annotateAnnotateADestructive
Write annotations (notes, comments, sticky notes, marked areas) into a local media file so they survive across sessions and are visible to humans in the file itself: real PDF annotation objects for a PDF, a sidecar VTT for video/audio, and a sidecar JSON for every supported media type. (写入批注 / 注释 / 评论 / 标注:PDF 原地写真正的批注对象;视频、音频写同名 .mmgrep.annotation.vtt;所有类型都写 .mmgrep.annotation.json 侧车。 中文触发词:批注、注释、评论、标注、annotate、comment、note。)
The sidecar ".mmgrep.annotation.json" is the single source of truth. The first write (no sidecar yet) imports every note-like annotation of a PDF — Text, FreeText, Square, Circle, Line, Polygon, PolyLine, Highlight, Underline, Squiggly, StrikeOut, Stamp, Caret and Ink, in page order — as n1…nN, including its raw geometry and original author, then applies this call. Link annotations, form widgets and popups are never imported or touched. Entries are anchored per media type:
PDF: page (1-based PHYSICAL page) + type ("text" = sticky note, "square" = rectangle area; "square" also needs rect {x,y,width,height} as fractions 0..1 of the page). Other annotation types cannot be created in this version, but imported ones are kept verbatim (including raw geometry) and can be updated (note) or removed.
video/audio: start (seconds, < duration) and optional end (default start + 5 s, clipped to the end of the media). No page, no rect, no type.
image: rect only (fractions of the image); no page, start/end or type.
every entry: note (plain text, 1..4000 characters, newlines allowed) and, for PDF, type.
Ids are n1, n2, …; new ids are max+1 and are never reused by this call; update keeps the id. One call can mix add / remove / update; they are applied remove → update → add, all-or-nothing: if anything is invalid, nothing is written and every problem is listed at once (unknown id, bad field, page/time out of range, duplicate ids, malformed entry, …).
The author is resolved once per call: the "author" argument, else the MCP client's name, else "MultiMediaGrep". The model name is never used. Imported annotations keep their original author; update never changes it.
WARNING — this tool modifies files:
A PDF is modified IN PLACE: annotation objects are written into the PDF file itself, no copy is made (the page content is not rewritten). Video and audio files are never modified — only the VTT is written.
Files are written NEXT TO the source: a .mmgrep. sidecar for every media type, plus *.mmgrep.annotation.vtt for video/audio. They are not written into the cache directory.
When a sidecar already exists, external edits to the file's own annotations are OVERWRITTEN on the next write. To change an annotation, use update; the sidecar wins over the file.
Protect your originals with git (or a backup) before calling this tool.
The result lists every written annotation in the same shape the read side uses ("n1 (p.12): note", "n1 (00:12:34–00:12:39): note"), plus the sidecar path, the embed target, and every check result. (批注 / 注释 / 评论 / 标注 / annotate / comment / note)
Inputs: path (required), add, author, remove, +1 more.
media_bookmarkBookmarkADestructive
Write bookmarks (an outline, chapters) into a local media file so they survive across sessions and are visible to humans in the file itself: the PDF outline for a PDF, a sidecar VTT for video/audio, and a sidecar JSON for every supported media type. (写入书签 / 目录 / 章节:PDF 改写文件自带目录;视频、音频写同名 .mmgrep.bookmark.vtt; 所有类型都写 .mmgrep.bookmark.json 侧车。中文触发词:书签、目录、章节、bookmark、outline、chapter。)
The sidecar ".mmgrep.bookmark.json" is the single source of truth. The first write (no sidecar yet) imports the file's own outline — PDF bookmarks, or video/audio chapter markers — as b1…bN in file order, then applies this call. New ids are max+1 and are never reused; update keeps the id. One call can mix add / remove / update; they are applied remove → update → add, all-or-nothing: if anything is invalid, nothing is written and every problem is listed at once (unknown id, bad field, page/time out of range, duplicate ids, malformed entry, …). An image has no native outline and gets the sidecar only; text and Office documents are not supported.
WARNING — this tool modifies files:
A PDF's outline is rewritten IN PLACE: the PDF file itself is modified in place, no copy is made.
Files are written NEXT TO the source: a .mmgrep. sidecar for every media type, plus *.mmgrep.bookmark.vtt for video/audio. They are not written into the cache directory.
When a sidecar already exists, external edits to the file's own outline are OVERWRITTEN on the next write.
Protect your originals with git (or a backup) before calling this tool.
After the write, media_probe / media_overview / media_locate show this outline (ids b1…bN) instead of the file's native one. The result contains the full outline, the sidecar path, the embed target, and every check result. (书签 / 目录 / 章节 / bookmark / outline / chapter)
Inputs: path (required), add, remove, update.
media_locateLocateARead-only
Find where something appears in a local media file and return its position: a timestamp (milliseconds and seconds) for a video or recording, a page number for a document. (定位:给一段文本或关键词,返回它在视频 / 录音里的时间戳,在文档里的页码。)
Use it when you know roughly what was said but not when ("where does it mention X"), or when you need to know which page a phrase is on. Matching ignores punctuation, spacing and case, and works for both Chinese and English. Long phrases match better than single characters.
For a video or recording, each match is a time window with a match score (0..1) and the matched line. For a document, each match is a page: the page number, how many times the query appears on that page (hits), and a short snippet around the first hit — never the whole page. Pages that matched but did not fit are listed as "N more pages match: ...". Read a full page with media_overview({ path, pages }).
For DOCUMENTS, ocr: "auto" (default) searches the PDF text layer where one exists and OCRs only pages without one; ocr: "force" re-recognizes every requested page with the OCR provider first, even pages that already have a text layer, and tags those matches (ocr). Use force when media_probe says "text layer: looks unreliable". Forced results are cached separately from auto. (文档:ocr: "force" = 强制重新 OCR / 重新识别文字层)
Text only by default — no image, no pixels spent. Add includeFrame: true to also get the picture at the best match (one image, roughly 1,900 vision tokens).
A transcript is required: the embedded subtitle track, a subtitle you select explicitly, or ASR configured via MMGREP_ASR_PROVIDER. Subtitles win over ASR. If none is available this tool says so instead of guessing. Use media_overview if you only need the visuals.
Each match is tagged with its source. A match whose source is "asr" came from machine transcription, so the wording may be wrong (names, numbers, jargon) — do not present it as the speaker's exact words.
The result also carries the same single outline as media_probe / media_overview in structuredContent.outline: the bookmark sidecar first, then the file's own PDF bookmarks or chapter markers. Its entries carry the ids media_bookmark accepts (b1…bN), and structuredContent.outline.sidecarPath / sidecarError point at the bookmark sidecar when one exists. Locate is read-only: it never creates or modifies a sidecar. (定位结果也带同一份目录:sidecar 优先,其次文件自带目录;line 首 b1…bN 可用于 media_bookmark)
It also searches the ANNOTATION text: the notes written next to the media (annotation sidecar ".mmgrep.annotation.json", or the file's own PDF annotations) are searched too, and annotation hits are reported in their own "annotation matches" section. They are marked as annotations (kind "annotation" plus the id, n1, n2, …) and are NEVER counted as media content hits — the match count above them counts transcript / page matches only. An annotation hit is a note somebody wrote; it is not something the media says. (定位也搜批注文本;批注命中单独成段,不计入媒体命中数)
Inputs: path (required), query (required), end, max, +7 more.
media_overviewOverviewARead-only
Build an overview of a local media file so you can read it without guessing. (生成媒体总览:视频出分镜宫格图与按时间对齐的文本清单;录音出带时间轴的转写;文档出页面缩略图与每页文字。)
What you get depends on the media:
video : one or more numbered storyboard grids (each cell carries its cell number) plus a time-aligned text index of the cells
audio : a timestamped transcript, segment by segment
document : a grid of page thumbnails plus page text, when a document parser is configured
For DOCUMENTS, ocr: "auto" (default) uses the PDF text layer where one exists and calls the OCR provider only for pages without one; ocr: "force" re-recognizes every requested page with the OCR provider, even pages that already have a text layer, and tags them (ocr). Use force when media_probe says "text layer: looks unreliable" or when the extracted text is visibly garbled. Forced results are cached separately from auto. (文档:ocr: "force" = 强制重新 OCR / 重新识别文字层)
Pass images: 0 for a text-only, zero-pixel overview: no grid is rendered and no frame is decoded (a video costs no ffmpeg work at all), so you get only the transcript — or the page text for a document — and the reported budget is all zeros. Use it when you only need the words.
TEXT SOURCE ORDER — subtitles first, ASR only as a fallback:
a subtitle you select explicitly (external subtitle file, track index or language code),
the subtitle track embedded in the container,
a subtitle file with the same base name in the same directory (a.srt next to a.mp4),
ASR — only when none of the above exists and an ASR provider is configured. Every segment is tagged with its source. subtitle = from a subtitle file/track; usually the author's words, but may be auto-generated: check names and numbers against the frame. asr = machine transcription: names, numbers and jargon are often wrong, so do NOT quote an ASR segment verbatim and do not present it as the speaker's exact words. visual = there is no text at all, only the picture.
The storyboard grid is for FINDING things, not for reading them: cells are small, so text inside a cell is usually unreadable. To read text on screen, call media_zoom on that moment and read the cropped single frame.
The full text is NOT inlined. It is written to disk, and the text of the result ends with a line like "full text: /abs/path/content.md (segments.jsonl, manifest.json in the same directory)". That absolute path is what you read: content.md is the readable full text, segments.jsonl has one segment per line, manifest.json records what was produced and how. Read those files section by section and summarise them in chunks yourself — do not ask for the same range again just to get more inline text. If the export could not be written, the line reads "full text: NOT written — " instead.
The inline text is a PREVIEW with three tiers. (1) WHOLE: the requested range fits, and the result says "complete text ... nothing omitted". (2) SKIM (documents only, when the document parser exposes block structure): every heading plus the first complete sentence of each paragraph, tables as "[table: caption]", with captions/footnotes/equations left out; each page is labelled like "[page 4 · skim: 12 of 210 sentences]". A skim built from the built-in heuristic blocks is best-effort — section structure may be wrong; read content.md for the authoritative text. (3) SAMPLED: complete units — document sentences, video/audio transcript segments — are sampled evenly, with a line like "preview: sampled 84 of 312 sentences across pages 1-28; not represented: 2-6, 8-12". In a skim that still does not fit, all headings are kept and sentences are sampled; only when even the headings do not fit are the headings sampled. Sampling never drops a whole page silently: pages that could not be represented are listed, and a single over-long unit is truncated with a trailing "…[+N chars]". Read content.md when you need the complete text.
VIEW — the view parameter controls the inline text (default "auto"). The result carries ONE outline from the first available source: the bookmark sidecar (".mmgrep.bookmark.json", written by media_bookmark) when one exists, otherwise the file's own outline (PDF bookmarks, or chapter markers in video/audio); a document without bookmarks may get an outline GENERATED FROM ITS HEADINGS when the range has at least 5 headings, labelled "outline: generated from headings" (best-effort). (1) "auto": if the requested range fits, you get the full text; if it does not fit and there is an outline, the outline comes first (ids b1…bN with page numbers for PDFs, timestamps for chapters) and the rest of the budget goes to the text preview; if there is no outline, you get the normal skim/sampled preview. (2) "outline": return only the outline — generated from headings for a document without bookmarks; for video/audio without chapters you get an explicit "no chapters" message and NOTHING is generated, because this server never invents an outline from content. In that case read the overview (or a start/end window) and write your own outline from the time-aligned text and storyboard grids. (3) "text": ignore the outline and force the text pipeline. The full outline (entries, source, sidecar path) is always in structuredContent.outline and, for an overview that writes an export, in manifest.json. When a sidecar exists its last outline line is "full detail: " — read that file for the complete bookmark detail. (目录:sidecar 优先,其次文件自带目录,最后才是 heading 生成;view:auto 放不下时先出目录;view:outline 只出目录;view:text 只出文本)
The result also carries the ANNOTATIONS of the requested range, right next to the page / time range they belong to: the annotation sidecar (".mmgrep.annotation.json", written by media_annotate) when one exists, otherwise the file's own annotations (PDF annotation objects). One annotation per line, in the same shape media_annotate writes ("n3 (p.12): note" for PDFs, "n3 (00:12:34–00:12:39): note" for video/audio). The line starts with the annotation id (n1, n2, …) — these are the ids media_annotate accepts. The author is in structuredContent.annotations only; the note text is shown as written. (批注按请求范围展示:行首 n1…nN 就是 media_annotate 的 id)
Always call media_probe first: it tells you how long the file is, whether a subtitle source exists, which transcript source applies, and which outline applies (bookmark sidecar first, then the file's own PDF bookmarks / video-audio chapters, or headings for a document), for free. Long media must be narrowed with start/end (video, audio) or pages (document) — one giant grid of a two-hour film gives you illegible cells and spoils the ending. Follow the continuation hint in the result to page through the rest.
Inputs: path (required), end, ocr, view, +9 more.
media_probeProbe mediaARead-only
Inspect a local media file — video, audio recording, or document — and report what it is: duration, resolution, page count, codec, available tracks (audio / subtitle / text layer), and whether transcription (ASR) and OCR are available for this file. (读取本地媒体的元信息:时长、分辨率、页数、编码、可用轨道(音轨 / 字幕轨 / 文本层), 以及能否做 ASR / OCR 的初判。)
Call this FIRST on any file you have not inspected yet. It costs no image tokens and decodes nothing. Then pick the right next tool:
video / audio -> media_overview to read it
document -> media_overview to see page thumbnails and per-page text
a specific moment or page -> media_zoom for a close-up
"where was X said / on which page" -> media_locate for a timestamp or a page number
For documents it also checks whether the text layer looks unreliable — U+FFFD replacement characters, private-use characters, isolated single letters, symbol noise — and, when it does, points at media_overview({ path, ocr: "force" }) to re-recognize the pages. (文档文字层看起来不可靠时会提示,可用 ocr: "force" 强制重新识别)
For audio it also reports a quick health check — average level, speech ratio and silence ratio — so you know whether the recording actually contains audible speech before you spend an ASR call on it. A recording that is effectively silent is reported as such and is never sent to ASR (no hallucinated transcript).
It also shows ONE outline for the file, from the first available source: the bookmark sidecar (".mmgrep.bookmark.json", written by media_bookmark) when one exists, otherwise the file's own outline — PDF bookmarks (PHYSICAL page numbers, the same 1-based numbers media_overview accepts) or chapter markers in video/audio (title + start/end timestamps). Every line carries a bookmark id ("b3 p.12 title" for PDFs, "b3 00:12:34 title" for video/audio) — these are the ids media_bookmark accepts. When a sidecar exists, the last outline line is "full detail: ": read that file for the complete outline. The text shows at most 2 levels / 60 entries; structuredContent.outline has every entry, the source and the sidecar path. Read-only: probing never creates or modifies a sidecar. (读取侧只展示一份目录:sidecar 优先,其次文件自带目录;行首 b1…bN 就是 media_bookmark 的 id)
It also reports the ANNOTATIONS of the file: how many there are and where they live — the annotation sidecar (".mmgrep.annotation.json", written by media_annotate) when one exists, otherwise the file's own annotations (PDF annotation objects). Probe reports the COUNT and the sidecar path only; the notes themselves are shown by media_overview, next to the page / time range they belong to ("n3 (p.12): note" for PDFs, "n3 (00:12:34–00:12:39): note" for video/audio). media_locate can also search the annotation text. Read-only: probing never creates or modifies a sidecar. (读取侧也展示批注:条数 + 侧车路径;note 本身在 overview 里按页 / 时间范围展示)
This server never downloads anything and never calls an LLM: you always pass a local absolute path, and it only prepares images and text for your own model to look at.
Inputs: path (required), subtitle, transcript.
media_zoomZoomARead-only
Get a close-up of one moment in a local video, or one page of a local document, returned as a single image. (取本地视频某一时刻的特写单帧,或本地文档某一页的整页 / 局部放大图。)
Use it after media_overview when one cell or page needs a closer look: "what exactly is on screen at 01:12?" "what does line 3 of page 7 say?"
For video, pass at (a single timestamp) or start/end (a window — the sharpest frame inside it is returned, and the exact timestamp is reported back). For a document, pass page, optionally with crop. crop is {x, y, width, height}, each a fraction 0..1 of the frame or page, origin top-left.
Keep it to one frame. It costs roughly 1,900 vision tokens — far more than a whole overview cell. Zoom the parts that matter instead of re-scanning the file.
This is the ONLY tool whose image is meant for reading. A storyboard cell is a few hundred pixels wide and the text inside it is unreadable; a cropped single frame at up to 1600 px is readable. So: locate in the overview grid, then zoom to read. The returned image is always source "visual" — what you read off it is your own reading of pixels, not a transcript, so treat it as an observation rather than a quotation.
Inputs: path (required), at, end, crop, +3 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 6 tools
Each tool occupies a distinct slot in the workflow: probe (metadata/health), overview (read content), locate (search text), zoom (read one frame/page), annotate (write notes), bookmark (write outline). The descriptions explicitly disambiguate the closest pair (overview grid = finding, zoom = reading) and warn against misselection. Overlap in secondary outputs (probe/overview/locate all surface the outline) does not create real selection ambiguity because primary purposes are clearly separated.
All six tools follow an identical media_<verb/noun> pattern with a shared prefix: media_probe, media_overview, media_locate, media_zoom, media_annotate, media_bookmark. No mixing of conventions or stray abbreviations.
Six tools is well-scoped for a local media inspection server: four read-side (probe, overview, locate, zoom) and two write-side (annotate, bookmark), each covering a genuinely different operation. No tool appears redundant or vestigial.
The surface covers the full lifecycle for its domain: inspect, read (text + visual), search, close-up, and persist bookmarks/annotations including add/update/remove. Minor gaps remain, such as no batch/multi-file operation and no dedicated way to discard or reset a sidecar wholesale, but these are workable via the existing remove operations.
Maintenance
Related MCP Connectors
PDF, image, video, OCR, screenshot, SQL, QR and text tools for agents. No API key, no signup.
Files to text for AI agents: audio/podcast transcription, OCR, video frames, screenshots, web to MD.
15 media & data tools for AI agents: search, transcribe, subtitles, voiceover, translate & more.
PDF and image tools for AI agents: compress, merge, split, OCR, sign. Runs locally, never uploads.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceProvides local, offline transcription, keyframe extraction, OCR, and pre-publish review of audio, video, and image files, enabling AI agents to see and hear media without cloud or API keys.59 npmApache 2.0
- AlicenseAqualityCmaintenanceLets AI agents inspect local or remote videos by reading metadata, detecting shots and cuts, extracting frames as images, and checking keyframes for seeking and streaming, all via local FFmpeg without an API key.51MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to inspect UI screen recordings as token-efficient image sheets with changed regions boxed, single-frame flicker flagged, and speech aligned to frames. Exposes watch_video and zoom_video tools so agents can pull dense keyframes across any time range or one sharp full-resolution frame, all processed locally.23 npmMIT
- AlicenseNot gradedqualityCmaintenanceEnables coding agents to ask focused questions about local screenshots, video, audio, and music and receive compact text reports on OCR, layout, timestamps, sounds, and music description. Media stays on the machine and raw files are never loaded into the agent's context, with support for cropping, interval sampling, and durable background jobs.MIT