process_media
Ingest a local video or audio file and transcribe its speech, extract keyframes, OCR on-screen text, and diarize speakers to make the media queryable.
Instructions
Ingest a LOCAL video or audio file and make it queryable: validates the file, transcribes speech locally (whisper), extracts scene-change keyframes, OCRs on-screen text, resolves the wall-clock start time, and (opt-in) labels who said what via local speaker diarization. Returns a compact summary (job_id, media info, wall_clock, transcript preview, speaker roster when diarized) — full data stays on disk and is served lazily by the other tools. Idempotent by content hash: re-calling on an already-processed file returns instantly. For MULTI-PERSON recordings (meetings, interviews, calls) diarize=true is part of a proper analysis — pass it even when the user only asks for a summary. When NOT to use: to re-fetch data you already processed (use the retrieval tools), or for URLs — local file paths only. Examples:
process_media(path="/Users/sam/Desktop/bug-repro.mov") — narrated screencast, defaults are right
meetings: model="large-v3-turbo" + vocabulary=<attendees, terms> + num_speakers=N — turbo's extra cost is trivial
process_media(path="/tmp/standup.m4a") — audio-only: transcript tools work, frame tools will error
process_media(path="/rec/panel.mov", diarize=true, num_speakers=4) — headcount known? ALWAYS pass it: best accuracy
job already processed + diarize=true → amend re-runs ONLY diarization (whisper reused; long files: minutes)
error mentions [diarization] → the extra is missing: install via uvx "talkthrough-mcp[diarization]"
know the attendees? process_media(path=..., vocabulary="Anastasia, Evgenii, OKR") — names+jargon survive STT
user: "analyze/summarize this meeting" → include diarize=true — speaker structure is not optional extra credit
noisy threshold roster (clusters ≫ people)? ASK your user for the real headcount, then re-run with num_speakers=N
cap_hit or sampling_interval_s in summary → for slide hunts raise TALKTHROUGH_MAX_FRAMES or use extract_frame
summary shows wall_clock=null → ask when recording started, re-call with recorded_at=... and force=true
transcript garbled or language_probability low → re-call with model="large-v3-turbo" (or language="ru") + force=true
after success, do NOT dump everything — continue with get_transcript / get_moment / search on the job_id
anti-example: frames from an already-processed job → get_frames(job_id=...), never process_media again
anti-example: YouTube/URL input → unsupported in v1; have the user download the file first
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | ||
| force | No | ||
| model | No | ||
| diarize | No | ||
| language | No | ||
| vocabulary | No | ||
| recorded_at | No | ||
| num_speakers | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||