Skip to main content
Glama
korovin-aa97

talkthrough-mcp

by korovin-aa97
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

talkthrough-mcp

ci license: MIT python PyPI

Quickstart · Tools · Benchmarks · FAQ · Troubleshooting · Changelog · Contributing

Don't write a bug report. Record it.

Give Claude Code or Codex a narrated .mov/.mp4 — or a public video link — and talkthrough turns it into searchable transcript, exact frames, OCR and wall-clock timestamps, locally — so your agent writes an evidence-backed issue draft or investigates the fix.

Also works for meetings, workshops, product demos, and production incidents.

Illustration of a /talkthrough:bug run: a recorded checkout bug is indexed locally, the evidence is found, and a ready-to-file issue draft is assembled

Illustration — an animation of the /talkthrough:bug storyline, not a screen capture: the recording is indexed locally (transcript · keyframes · OCR · wall-clock), the evidence checkpoint is assembled, and the draft is ready to file with your own tracker tooling (gh, Jira, a GitHub MCP server — talkthrough itself never leaves your machine and never posts anything).

Real runs, no animation:

  • ▶ Watch the demo with sound (1:18) — an unedited session: a narrated recording goes in, a ready-to-file bug-report.md comes out.

  • Silent recording → issue draft — the whole thing as files you can re-run: a Playwright-recorded, audio-free .mp4 and the unedited agent output. Every number is reproducible — processing that file gives job_id 8703a66bbe77a7d0 (the job id is the sha256 prefix, so shasum -a 256 predicts it), 17 keyframes / 4 unique, and 0 transcript segments because there is no audio track.

Quickstart

One command, no system dependencies: ffmpeg falls back to a bundled build, OCR is pip-only, and whisper models download themselves on first use. The only prerequisite is uv (brew install uv or curl -LsSf https://astral.sh/uv/install.sh | sh).

Cold setup has two separate stages: uvx first resolves a compatible Python and the pinned server environment, then the first process_media downloads any missing media/model assets. A plugin update can create a new environment and, without system ffmpeg, fetch the ~80 MB bundled ffmpeg again; shared Whisper/OCR/diarization caches and warm, network-free jobs remain reusable. See Troubleshooting.

Install in Cursor Install in VS Code Install in VS Code Insiders Add to LM Studio Add to Kiro

Claude Code

Two install paths — pick one, not both (the plugin already includes the server; installing both would register it twice):

Server only — the 9 tools + 6 prompts, and nothing else on your system. Choose this for a minimal setup, or when you manage MCP servers yourself across several clients:

claude mcp add -s user talkthrough -- uvx --python ">=3.11,<3.14" "talkthrough-mcp[diarization,url]"

Full plugin — the same server, plus native slash commands (/talkthrough:bug, /talkthrough:triage-recording, …) that handle the ceremony for you, a ready-made triage subagent, and an agent skill that teaches Claude the workflow. Choose this for the best out-of-the-box experience:

/plugin marketplace add korovin-aa97/talkthrough-mcp
/plugin install talkthrough@talkthrough

Every other MCP client

claude_desktop_config.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "--python",
        ">=3.11,<3.14",
        "talkthrough-mcp[diarization,url]"
      ]
    }
  }
}

More: integrations/claude-desktop/

~/.cursor/mcp.json (or project .cursor/mcp.json):

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "--python",
        ">=3.11,<3.14",
        "talkthrough-mcp[diarization,url]"
      ]
    }
  }
}

More: integrations/cursor/

~/.codex/config.toml (or project-scoped .codex/config.toml in trusted projects):

[mcp_servers.talkthrough]
command = "uvx"
args = ["--python", ">=3.11,<3.14", "talkthrough-mcp[diarization,url]"]

More: integrations/codex/

~/.gemini/settings.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "--python",
        ">=3.11,<3.14",
        "talkthrough-mcp[diarization,url]"
      ]
    }
  }
}

More: integrations/gemini-cli/

cline_mcp_settings.json (via MCP Servers UI):

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "--python",
        ">=3.11,<3.14",
        "talkthrough-mcp[diarization,url]"
      ]
    }
  }
}

More: integrations/cline/

~/.openclaw/openclaw.json:

{
  "mcp": {
    "servers": {
      "talkthrough": {
        "command": "uvx",
        "args": [
          "--python",
          ">=3.11,<3.14",
          "talkthrough-mcp[diarization,url]"
        ]
      }
    }
  }
}

More: integrations/openclaw/

opencode.json (project) or ~/.config/opencode/opencode.json:

{
  "mcp": {
    "talkthrough": {
      "type": "local",
      "command": [
        "uvx",
        "--python",
        ">=3.11,<3.14",
        "talkthrough-mcp[diarization,url]"
      ],
      "enabled": true
    }
  }
}

More: integrations/opencode/

~/.config/goose/config.yaml:

extensions:
  talkthrough:
    enabled: true
    type: stdio
    cmd: uvx
    args: ["--python", ">=3.11,<3.14", "talkthrough-mcp[diarization,url]"]

More: integrations/goose/

~/.copilot/mcp-config.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "--python",
        ">=3.11,<3.14",
        "talkthrough-mcp[diarization,url]"
      ]
    }
  }
}

More: integrations/copilot-cli/

~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "--python",
        ">=3.11,<3.14",
        "talkthrough-mcp[diarization,url]"
      ]
    }
  }
}

More: integrations/windsurf/

settings.json (Zed):

{
  "context_servers": {
    "talkthrough": {
      "source": "custom",
      "command": {
        "path": "uvx",
        "args": [
          "--python",
          ">=3.11,<3.14",
          "talkthrough-mcp[diarization,url]"
        ]
      }
    }
  }
}

More: integrations/zed/

Any other MCP stdio client uses the same server command: uvx --python ">=3.11,<3.14" "talkthrough-mcp[diarization,url]". Per-engine folders with exactly these snippets plus verification steps live in integrations/; agents can self-install via llms-install.md.

Who said what (speaker diarization) — included in the configs above

Multi-person recordings (meetings, interviews, panels) can carry S1/S2/… speaker labels. Every install button, snippet, and the plugin above already ship the [diarization] engine, so asking your agent "who said what" just works — diarization itself still runs only when requested per call (process_media(path=..., diarize=true, num_speakers=<count if known>)), and its models download once on first use.

Prefer the minimal server without the diarization engine? Use uvx --python ">=3.11,<3.14" talkthrough-mcp as the command instead (the MCP registry entry also resolves to this lean form) — an explicit diarize=true will then answer with the one-line install fix. Details in Speakers.

Upgrading from 0.3.x

Regenerated configs and the plugin carry [diarization,url]. A config you wrote by hand for 0.3.x — uvx --python ">=3.11,<3.14" talkthrough-mcp, or a pin without the url extra — upgrades the server in place and lists the new process_url tool, but only direct https:// media links work until the extra is there: YouTube and video pages answer with the one-line install fix. Add url to your command (uvx --python ">=3.11,<3.14" "talkthrough-mcp[diarization,url]"), restart the client, and check with talkthrough-mcp --version (0.4.1+), which names the extras the environment has; the server logs the same line to stderr at every start, so your client's MCP log shows it too.

Local checkout (development)

git clone https://github.com/korovin-aa97/talkthrough-mcp
claude mcp add talkthrough -- uv run --directory /path/to/talkthrough-mcp talkthrough-mcp

Then, in your agent:

Process ~/Desktop/recording.mov and triage it — or just invoke the triage-recording server prompt.

Related MCP server: pincushion-mcp

Tools

Tool

What it does

process_media(path, recorded_at?, vocabulary?, language?, model?, diarize?, num_speakers?, force?)

Ingest a video/audio file: local STT, keyframes, OCR, wall-clock, opt-in speaker labels. Returns a compact summary. Idempotent by content hash — re-calls are instant; diarize=true on a processed job adds speakers without re-transcribing.

process_url(url, recorded_at?, vocabulary?, language?, model?, diarize?, num_speakers?, refresh?, force?)

The one network tool: download one public video/audio URL once (a direct https:// media link, one YouTube video, or any public video page yt-dlp can read — with the [url] extra), keep the source inside the job, then run the same local pipeline. Same-URL re-calls serve the stored job without network unless refresh=true; force=true rebuilds from the kept source; the raw URL is never stored.

get_transcript(job_id, start_ms?, end_ms?, format?)

Paginated transcript as segments, text, or srt (speaker-prefixed when diarized, plus a roster header); truncation returns next_start_ms.

get_frames(job_id, at_ms? | start_ms?+end_ms?, max_frames?, include_duplicates?)

Keyframe images nearest a timestamp or evenly thinned across a range (unique frames by default, max 6/call); each frame names its absolute path.

get_moment(job_id, start_ms, end_ms)

The "one remark" bundle: transcript slice + up to 3 frames + their OCR text + wall-clock range (+ speakers_in_range when diarized).

search(job_id, query, speaker?, match_mode?)

Substring search over the transcript AND on-screen OCR text. all_words remains the default; any_word broadens lexical recall. Hits carry t_ms/t_wall, frame refs, and the speaker when diarized. The optional filter accepts a raw label or saved name.

label_speakers(job_id, labels, evidence?)

Atomically persist verified names for anonymous speaker labels. Raw S1/S2 labels remain canonical; blank/null removes a name.

extract_frame(job_id, at_ms, crop?)

Exact-timestamp full-resolution re-extract from the source video (optional crop) when keyframes miss the instant; returns the file's absolute path. URL jobs decode their kept source — no network.

list_jobs()

Recent processed recordings with source paths, durations, wall-clock starts, counts, speaker counts when diarized, and the provider/id for URL jobs.

Every tool description ships 10+ usage examples, so agents pick the right tool without extra prompting.

Server prompts (slash commands in MCP clients)

Prompt

Workflow

bug

One recording → evidence-backed GitHub issue draft (silent, narration-free recordings work too)

triage-recording

Narrated screencast → precise findings JSON (bug/feature/question routing, frame evidence)

spec-from-workshop

Recorded workshop → structured spec with quoted decisions and open questions

backlog-from-demo

Product demo → prioritized backlog with timestamped evidence

meeting-actions

Meeting audio → action items, decisions, open questions

correlate-with-logs

Recording remarks ↔ system logs via wall-clock windows

The same prompts live as plain files in examples/prompts/ if your client doesn't surface MCP prompts. The findings contract used by triage-recording is examples/output-contract.schema.json.

Works as a skill too (no MCP required)

The same workflow ships as a cross-engine Agent Skill at .agents/skills/talkthrough/ — Claude Code, Codex CLI ($talkthrough), Cursor, Copilot, Gemini CLI, Goose and other SKILL.md-compatible tools read it. Agents without MCP wiring can drive the CLI directly: talkthrough-mcp process recording.mov --json prints the same summary the MCP tool returns, and the job store is shared either way.

Wall-clock anchoring

Every timestamped result carries both t_ms (video-relative) and t_wall (ISO 8601 real time) once the recording start is known. Resolution ladder:

  1. recorded_at parameter (agent/user override) → confidence exact

  2. QuickTime com.apple.quicktime.creationdate tag, carries the local timezone (QuickTime Player recordings; ⌘⇧5 wrote it before macOS 26) → high

  3. Container creation_time tag (UTC) → medium — macOS 26+ ⌘⇧5/ReplayKit screen recordings land here (no creationdate tag anymore); pass recorded_at= when local-tz t_wall matters

  4. File mtime minus duration (recorders finalize files at recording END) → low

  5. Nothing → tools still work with relative t_ms only

Why it matters: "the upload spinner froze here" becomes a ±30 s grep window in your server logs.

Speakers (optional diarization)

With the [diarization] extra installed (included in every generated config — see Quickstart), process_media(diarize=true) labels who said what — locally, like everything else here (sherpa-onnx runtime, no torch, no accounts, no GPU):

  • Speakers become S1, S2, … in order of first appearance; new diarized jobs split speaker changes at word boundaries, while old jobs continue to report honest segment-level precision. Every transcript segment gets a speaker, and the tools surface it everywhere — roster with talk time in get_transcript, speakers_in_range in get_moment, speaker on search hits, S1: prefixes in the text/SRT formats, a speaker count in list_jobs.

  • Know the headcount? Pass num_speakers. Clustering toward an exact k removes the main failure mode of unknown-count mode (similar voices merging or one voice splitting). It is a target, not a guarantee: the clusterer can converge on fewer clusters than k, and a re-run that changed nothing says so in the payload (labels_changed: false). Agents are instructed to pass the headcount via the tool guidance; do the same in your own calls.

  • Already processed a recording? Calling process_media(diarize=true) on it re-runs only diarization — whisper is not re-run, and labels land in the existing job. Same for changing num_speakers. The diarization stage itself still re-scans the full audio: minutes on long recordings.

  • Full rebuilds keep named jobs safe. If a job has active or pending names, force=true also requires diarize=true; otherwise the call refuses before changing stored data. A successful force rebuilds in staging and moves every previous identity to pending review against the fresh roster. Any processing or commit failure leaves the prior manifest and frames intact.

  • Labels start anonymous. After checking self-introductions, vocatives, or video evidence, call label_speakers to preserve a verified mapping such as S1 → "Alice" across sessions. The roster can expose bounded OCR name_candidates, but those are raw hints and are never saved automatically. The raw label remains present beside speaker_name. If a later diarization amend changes the labels, verified names stop being active and move to bounded speaker_names_pending_review evidence instead of being silently lost. Re-check the current roster and explicitly confirm or remove each affected label with label_speakers.

  • Video jobs produced before 0.3.1 keep their original flat OCR and can return name_candidates_note to explain why hints are absent. They remain fully readable without migration; force=true, diarize=true regenerates line-aware OCR while preserving saved identities for review.

Models download once (~47 MB total) from pinned, checksum-verified URLs into ~/.talkthrough/models/; warm runs are zero-network like the rest of the pipeline. Speed on an M-series CPU (4 threads): a 26-minute meeting diarizes in about 2 minutes (RTF ≈ 0.08), on top of the transcription time. Memory: expect on the order of 1–1.5 GB peak RSS while an hour-plus meeting is being diarized (measured on a real 73-minute recording); it is released when the stage completes.

Role

Model

Download

Weights license

Segmentation

pyannote segmentation-3.0 (ONNX export by k2-fsa)

7 MB

MIT

Embedding (default)

NeMo en_titanet_small

40 MB

Apache-2.0 (per its NGC model card, the NeMo Toolkit license)

Embedding (alt)

WeSpeaker en_voxceleb_resnet34_LM

27 MB

CC-BY-4.0

Embedding (alt)

3D-Speaker campplus_sv_en_voxceleb_16k

30 MB

Apache-2.0

The default won a real-meeting accept-eval (RU/EN/ES + a 3-speaker 26-minute meeting): it was the only candidate to isolate all three real voices at num_speakers=3, at 2× the speed of the runner-up.

Pick an alternate embedding model (or point at your own .onnx file for offline machines) via TALKTHROUGH_DIARIZATION_EMB_MODEL; tune the unknown-count sensitivity via TALKTHROUGH_DIARIZATION_THRESHOLD (see docs/TROUBLESHOOTING.md). Honest quality notes live in Limitations.

Privacy

Everything runs locally: your recordings never leave your machine, speech is transcribed by a local whisper model, OCR and speaker diarization are local ONNX inference, and there is no telemetry. For local files the only network access is one-time tool/model downloads (ffmpeg build, whisper model, OCR models, diarization models — the latter pinned by URL + sha256). The one deliberate exception is process_url: it downloads the public source you name from its provider or CDN, once, and nothing else — no media ever goes up, no cloud STT or LLM is called, and after that download every tool on the job is network-free again. The raw URL (which may carry signed tokens) is not stored: the job keeps a hash, the public provider id or host and a bounded title. Diarization keeps no voiceprint database: voice embeddings live only in process memory, and only anonymous turn labels (S1/S2) land on disk. Your agent sees only the payloads the MCP tools return (text and selected frames) in your existing session; talkthrough itself makes no LLM calls.

Languages

Narration in any of Whisper's ~99 languages works: the language is auto-detected per recording, and the summary reports both language and language_probability so agents can tell a confident detection from a shaky one (silence or music at the start can fool the detector — pin it with language="ru" and force=true when that happens). Speaker diarization is acoustic — it fingerprints voices, not words — so it is language-independent and works across all of those languages unchanged.

Pick the model for your languages — per call (model= parameter, agents do this themselves when a transcript comes back garbled) or as the server default (TALKTHROUGH_WHISPER_MODEL):

Model

Size

Best for

small (default)

464 MB

English and major-language narration on CPU

large-v3-turbo

~1.5 GB

recommended for non-English — near-large quality at near-small speed

medium

~1.5 GB

conservative alternative to turbo

tiny / base

75–145 MB

quick drafts, CI

*.en variants

—

English-only, slightly faster/better for EN

Tips that work in every language: pass product names via vocabulary="Term1, Term2" (biases the decoder so jargon survives), and note that the workflow prompts instruct agents to write digests in the narrator's language while keeping quotes verbatim — the server never translates (exact quotes are evidence; translation is the agent's job).

On-screen text (OCR) defaults to RapidOCR's Latin + Chinese models. For other scripts set TALKTHROUGH_OCR_LANG to your language — ru/uk (→ the eslav pack), ja, ko, ar, hi, el, th, or any RapidOCR pack name like cyrillic — and reprocess with force=true; the matching recognition model downloads once. Spoken-language support is unaffected either way.

Configuration

Env var

Default

Meaning

TALKTHROUGH_WHISPER_MODEL

small

default whisper model (tiny/base/small/medium/large-v3/large-v3-turbo); the model tool param overrides per call

TALKTHROUGH_OCR

on

set off to skip OCR

TALKTHROUGH_OCR_LANG

Latin+Chinese

recognition script for on-screen text: a language code (ru, ja, ko, ar, hi, …) or a RapidOCR pack name (eslav, cyrillic, latin, …); the model downloads once

TALKTHROUGH_OCR_PARAMS

—

advanced: JSON object of raw RapidOCR params merged over the derived ones, e.g. {"Rec.lang_type": "cyrillic"}

TALKTHROUGH_DIARIZE

off

set on to diarize by default (needs the [diarization] extra; degrades with a warning without it); an explicit diarize tool param always wins

TALKTHROUGH_DIARIZATION_THRESHOLD

0.5

clustering sensitivity when num_speakers is unknown: fewer speakers than expected → lower it; more → raise it

TALKTHROUGH_DIARIZATION_SEG_MODEL

pyannote-segmentation-3-0

segmentation model: allowlist name or a path to a local .onnx (offline preseed)

TALKTHROUGH_DIARIZATION_EMB_MODEL

nemo_en_titanet_small

embedding model: allowlist name (see Speakers) or a local .onnx path

TALKTHROUGH_DIARIZATION_THREADS

min(4, cpus)

ONNX threads for both diarization models

TALKTHROUGH_MAX_SECONDS

7200

max media duration (also checked against provider metadata before a process_url download)

TALKTHROUGH_MAX_FRAMES

600

keyframe budget per job, spread across the whole duration (the 1 s selection floor auto-grows to duration/budget on long recordings)

TALKTHROUGH_MAX_DOWNLOAD_BYTES

2147483648 (2 GiB)

hard cap for one process_url download, enforced before and during the transfer

TALKTHROUGH_HOME

~/.talkthrough

job store root (URL jobs keep their downloaded source under jobs/<id>/source/)

CLI

The pipeline is also a CLI — useful for pre-processing long recordings outside an agent session (the store is content-addressed, so the agent then queries the same job instantly):

talkthrough-mcp process ~/Videos/long-session.mov   # prints the summary
talkthrough-mcp process demo.mov --json             # machine-readable
talkthrough-mcp process sync.m4a --diarize --num-speakers 3   # who said what
talkthrough-mcp process-url "https://youtu.be/nHfGfEiVdE8"    # one public URL, downloaded once
talkthrough-mcp gc --keep-days 30                   # clean the job store (sources go with their jobs)
talkthrough-mcp serve                               # stdio MCP server (default)
talkthrough-mcp --version                           # package version + which extras this environment has

--json keeps stdout machine-readable on failure too (0.4.1), including a missing argument or unknown option: the process exits with code 2, stderr carries the human error: … line, and stdout carries one JSON document, {"error": {"type": "UnsupportedUrlError", "message": "…"}}. --version also says which optional extras the environment has — the quickest check when a hand-written config launches the minimal server (a launcher without the url extra) that advertises process_url but can only read direct media links; the server logs the same line to stderr at every start.

First run notes: missing system ffmpeg triggers a one-time static-ffmpeg download; the first transcription downloads the whisper model (~460 MB for small); both are cached. After that, expect roughly 3× faster than real time on an Apple-Silicon CPU with the default model, OCR included (a 2-minute clip processes in ~40 s) — and instant re-runs on the same file. Progress streams as MCP progress notifications, and the CLI prints stage lines. More: docs/TROUBLESHOOTING.md.

Windows

CI runs lint, the unit suite, a full CLI smoke, and a diarize smoke on windows-latest (static-ffmpeg Windows build, whisper tiny transcription, OCR, the instant idempotent re-run, and a speaker-roster assert through the native sherpa-onnx stack). Notes: the per-job lock always serializes threads; POSIX also uses fcntl for cross-process locking. Quote paths with spaces (uv run talkthrough-mcp process "C:\Videos\Screen Recording.mp4"). If something breaks, please open an issue.

Supported inputs

Video: .mov .mp4 .webm .mkv .ogv — audio-only: .m4a .mp3 .wav .ogg .flac (transcript tools only; frame tools explain why they're unavailable).

URLs (via process_url, since 0.4.0): a direct https:// link to one of those media files; one public YouTube video (watch, youtu.be, shorts, a completed live); or any public video page yt-dlp can read — public Instagram reels, TikTok, Wikimedia Commons file pages (each verified on release day), the rest of yt-dlp's ~1800 site extractors and pages with a plain HTML5/HLS player (the [url] extra, which the generated configs above already carry, brings yt-dlp). Always anonymous: a site that demands a sign-in from anonymous clients (Vimeo does, with this yt-dlp) is refused with the reason. The source is downloaded once, kept inside the job, and never re-fetched for later questions. Not supported: playlists, channels, active live streams, private, members-only, age-restricted or DRM-protected videos, cookies or logins — a site that hides a video behind a login or a bot wall answers with a clear refusal, not a workaround (Instagram in particular rate-limits anonymous access). You are responsible for having the right to download and process what you point it at; talkthrough does not bypass any restriction.

Limitations

Honest edges, so you can decide fast:

  • Speaker labels are word-level and opt-in on new jobs. Each Whisper word is assigned by maximum overlap with the diarization turns, so fast exchanges split without losing the raw label. Old jobs remain readable and honestly report attribution_precision="segment"; reprocess with force=true to add word timings. Sub-second interjections ("yeah", "mhm") can still be absorbed when the diarization engine does not detect a separate turn, and heavy crosstalk degrades clustering (the segmentation model tracks at most 2 simultaneous voices). Quality is pyannote-3.x-generation. The comfort zone without hints is roughly 2–8 speakers; pass num_speakers whenever the headcount is known — it removes the worst failure mode at any size, and it is the way to go for large meetings (10+).

  • URL ingestion covers one public video at a time, without logins. Direct HTTPS media links, single public YouTube videos and public video pages yt-dlp can read; playlists, channels, live streams, gated or DRM content and anything behind a login or a bot wall are refused with a reason. Sites change; a page that worked yesterday can need a newer yt-dlp tomorrow. A provider's upload date is not a recording time, so URL jobs have wall_clock: null unless you pass recorded_at.

  • Memory: budget about 2 GB for a cold run. Whisper, the OCR models and the frame pass live in one process. A 78-second video on tiny with OCR peaked at 1.6 GB RSS during the 0.4.0 release QA (download included); the default small model needs more, larger models proportionally so. An 8 GB laptop copes; on anything tighter keep the model small or run the CLI ahead of the agent session.

  • Keyframes + transcript, not motion analysis. A glitch between scene changes can be invisible in the frame set; extract_frame re-checks any instant, but frame-by-frame motion reasoning is your multimodal model's job.

  • STT quality tracks the model you pick. The default small favors speed; non-English narration wants model="large-v3-turbo" (see Languages).

  • OCR reads crisp UI text well; tiny or low-contrast print is best-effort.

  • Wall-clock confidence depends on recorder metadata — worst case pass recorded_at= (see the ladder above).

  • Windows caveats — POSIX lock degrades to a no-op; see the Windows section above.

How it compares

talkthrough

cloud recorder SaaS

meeting notetakers

typical video-analyzer MCPs

Runs fully locally

✅

❌

❌

varies

Any local video/audio file

✅

browser/app captures

meetings only

✅

Public video URL: downloaded once, kept with the job, analyzed locally

✅

n/a

n/a

temp download, often cloud analysis

Wall-clock anchoring (log correlation)

✅

❌

❌

❌

Who-said-what speaker labels

✅ local, opt-in

some

✅ cloud

❌

Ships agent workflows (prompts, skill, findings contract)

✅

❌

❌

❌

OCR of on-screen text, searchable

✅

some

❌

rare

FAQ

Why not just upload the video to a multimodal model (e.g. Gemini)? For a short, non-sensitive clip — do that. The trade-offs appear with length and sensitivity: an hour of screen recording costs on the order of a million tokens per question, the file leaves your machine, and you still can't map a remark to 14:32:07 UTC to grep your server logs. talkthrough indexes once, locally, then answers any number of follow-ups from the index.

Why not screenpipe? Different job. screenpipe is an always-on recorder of your machine going forward (commercial license). It can't open the .mov a teammate or customer just sent you. talkthrough analyzes any file it's handed — the two compose fine.

There are agent skills that "watch" videos. Why a server with an index? Watch-style skills push a budgeted frame dump into the context window (and go sparse on long videos), often call cloud STT for the audio, and keep nothing. talkthrough builds a persistent local index — transcript + OCR, full-text searchable — retrieves exact frames lazily, anchors everything to wall-clock time, and answers the next question without reprocessing.

I use Jam for bug reports — do I need this? Keep Jam for browser bugs: console+network captured at record time is great evidence. talkthrough covers what a browser extension can't — desktop apps, mobile screencasts, ops incidents, meetings, any file — with no account, and correlates with server-side logs via wall-clock time.

Which agent model do I need to drive this? For v0.4.0 the six model configs below ran 61 isolated behaviour cells on URL ingestion (YouTube, a TikTok page, a speechless Instagram reel, playlist refusal, the missing wall clock) and the 0.3.2 integrity fixes; every cell passes on every runner after two product fixes the first attempts exposed (Codex needs a per-tool approval for the open-world process_url, and the server now says itself when a URL job has no wall clock). For v0.3.0 we ran 210 isolated agent cells across 6 model configs (Claude haiku/sonnet/opus, Codex gpt-5.5 at two reasoning efforts, and gpt-5.4-mini) and 35 logical scenarios on 5 real recordings plus safety and speaker-label fixtures. All 102 LLM-judged full-grid results and every mechanical zero were manually audited; the 30 new speaker behavior runs passed, and old-server control left 0 release-caused regressions. This is a model-drift snapshot, not a leaderboard: see the current matrix in docs/MODEL-NOTES.md. The chart and narrative in benchmarks/ remain the historical v0.2.0 snapshot.

Can't I just script ffmpeg + whisper myself? Yes — that's exactly this pipeline. What you'd be rebuilding: scene-change detection with perceptual dedup, OCR, transcript+OCR search, the wall-clock ladder, MCP tools with embedded usage examples, six workflow prompts, and a findings contract. One uvx command instead of an afternoon of glue.

Is it really local? What leaves my machine? Nothing goes up, ever. For local files the network is used only for one-time downloads (ffmpeg build, whisper/OCR/diarization models). process_url is the single tool that talks to the network at runtime, and only down: it fetches the public source you named, once. No telemetry. See Privacy — and SECURITY.md treats a violation of this promise as a vulnerability.

For agents & tooling

Machine-readable entry points, so AI agents can install and use this server without a human reading docs:

Roadmap

cloud STT · embeddings/semantic search · hosted/remote mode · .mcpb bundle · whisper.cpp backend

License

MIT

Available Tools

9 tools
extract_frameA
Idempotent

Re-extract ONE frame at an exact timestamp from the ORIGINAL source video at native resolution, with an optional crop {x, y, w, h} in source pixels. Use when the stored keyframes miss the instant (they capture scene changes + a 1 fps floor) or when you need full-resolution detail. Slower than get_frames — it decodes the source file, which must still exist at its recorded path (URL jobs keep their downloaded source inside the job, no network). When NOT to use: normal browsing — get_frames serves stored keyframes instantly without touching the source. Examples:

  • keyframes sit at 12:31 and 12:38 but the flash happened at 12:34.5 → extract_frame(job_id, at_ms=754500)

  • extract_frame(job_id="...", at_ms=754500, crop={"x":800,"y":40,"w":400,"h":120}) — zoom into the toast text

  • tiny UI text unreadable in the 1568px keyframe → extract_frame at the same ms for native resolution

  • verify a one-frame glitch: extract_frame at 12300, 12400, 12500 and compare

  • OCR missed small text → extract_frame with a tight crop, then read the returned image

  • crop coordinates are SOURCE pixels (a Retina screen recording may be 2940x1912) — not keyframe scale

  • response JSON carries "path" (absolute) — "save this screenshot next to my docs" = copy from path yourself

  • source file moved or deleted → clear error; stored keyframes via get_frames still work

  • audio-only job → always errors: there is no video stream to decode

  • anti-example: "show me around 5:00" → get_frames(job_id, at_ms=300000); extract_frame is for exact instants

  • anti-example: scanning a range frame by frame → get_frames(start_ms, end_ms) first, refine once after

ParametersJSON Schema
NameRequiredDescriptionDefault
cropNo
at_msYes
job_idYes

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations only provide idempotentHint, but the description discloses much more: performance cost, source-file dependency, URL-job storage behavior, audio-only failure, error behavior when source is moved or deleted, and the response JSON's 'path' field. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Long but dense; every block — use cases, not-to-use guidance, examples, anti-examples, caveats — adds operational value. The core purpose is front-loaded, and the detail is justified by the tool's subtle relationship to get_frames.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with no output schema, the description covers inputs, failure modes, performance trade-offs, and result handling sufficiently for an agent to select and invoke it correctly. The 'copy from path yourself' note closes a real usage ambiguity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description carries the full burden and succeeds: at_ms is tied to exact timestamps with concrete ms examples, crop is fully defined with {x, y, w, h} and source-pixel semantics, and job_id is shown in call examples.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb ('Re-extract') and precise scope: one frame, exact timestamp, original source video, native resolution, optional crop. The repeated contrast with get_frames makes it unmistakable which sibling tool is meant.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use it (keyframes miss the instant, need full-resolution detail) and when not to use it (normal browsing, range scanning), with anti-examples routing to get_frames. This is strong, unambiguous selection guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_framesA
Read-onlyIdempotent

Fetch stored keyframe images (JPEG, <=1568px wide) as MCP image content: the frames nearest to at_ms, OR unique frames across [start_ms, end_ms] evenly thinned to max_frames. Serves unique frames by default (near-duplicates from static scenes are filtered); hard cap 6 images per call. When NOT to use: exact instants between keyframes or native-resolution detail (use extract_frame), or finding on-screen text (use search — OCR text is indexed). Examples:

  • get_frames(job_id="...", at_ms=83500) — what was on screen when the remark at 1:23.5 was spoken

  • get_frames(job_id="...", at_ms=83500, max_frames=2) — tighter context, fewer tokens

  • get_frames(job_id="...", start_ms=0, end_ms=600000, max_frames=6) — overview strip of the first 10 min

  • get_frames(job_id="...", start_ms=290000, end_ms=310000, include_duplicates=true) — every capture near 5:00

  • transcript hit at t_ms=421500 → get_frames(job_id, at_ms=421500) for the visual evidence

  • walking a demo scene by scene → one ranged call per scene beats one giant range

  • frame files are named by video-ms (t00083500.jpg ↔ t_ms 83500) — stable refs for findings

  • valid_from_ms/valid_to_ms on each frame = when the screen looked like this — check the span covers your moment

  • keep max_frames at 2-4 unless you are truly comparing scenes; images are token-expensive

  • every frame entry carries "path" (absolute) — save/copy the image elsewhere with your own file tools

  • audio-only job → this tool errors by design; use get_transcript / get_moment instead

  • anti-example: need EXACTLY 12:34.500 between two keyframes → extract_frame(job_id, at_ms=754500)

  • anti-example: "find the screen with the red error banner" → search(job_id, "error") first, then jump

ParametersJSON Schema
NameRequiredDescriptionDefault
at_msNo
end_msNo
job_idYes
start_msNo
max_framesNo
include_duplicatesNo

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only/idempotent safe, but the description adds substantial behavioral detail: hard cap of 6 images, deduplication of near-duplicates, error on audio-only jobs, file naming convention, valid_from_ms/valid_to_ms semantics, and path field disclosure. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Long but efficiently structured: core behavior, when-not, examples, anti-examples, and field tips. Every bullet adds actionable info; front-loaded with the primary fetch behavior, and the length is justified by the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description compensates by explaining return content (MCP images, path, valid_from/valid_to), error cases, and practical usage limits. Complete for an agent to select and invoke correctly without additional context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 0% description coverage, so description must and does explain semantics. It clarifies at_ms (nearest), start/end_ms (range), max_frames (thinning), include_duplicates (every capture), and job_id via examples, fully compensating for the schema gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('Fetch') and resource ('stored keyframe images') plus format constraints (JPEG, <=1568px). It clearly differentiates from siblings by naming extract_frame for exact instants/native resolution and search for text, and get_transcript/get_moment for audio-only jobs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly lists 'When NOT to use' with alternatives (extract_frame, search) and provides anti-examples. Examples show when to use, including ranged vs at_ms, include_duplicates, and transcript-driven lookups, giving clear decision guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_momentA
Read-onlyIdempotent

The "one remark" evidence bundle: transcript slice + up to 3 unique frames + their OCR text + the wall-clock range for [start_ms, end_ms], in a single call. This is the workhorse for triage: one call per finding gives you the quote, the screenshot, and the on-screen text. When NOT to use: broad exploration (get_transcript / get_frames) or keyword lookup (search). Examples:

  • get_moment(job_id="...", start_ms=83000, end_ms=97000) — full evidence for the remark at 1:23-1:37

  • segment seq 12 spans t0_ms=83210, t1_ms=96800 → get_moment(job_id, 83210, 96800)

  • pad ±2000 ms around the spoken range — narrators react to things already on screen

  • triage loop: for each candidate finding, exactly one get_moment call → quote + frame + OCR

  • user: "what was I showing when I said 'this button is broken'?" → search first, then get_moment at the hit

  • opening context of a meeting: get_moment(job_id, 0, 15000)

  • response includes the t_wall range when known → quote it in bug reports for log correlation

  • diarized job → speakers_in_range + speaker on each segment: who is talking in this window, at a glance

  • frame entries carry "path" (absolute) — copy the screenshot elsewhere with your own file tools

  • "was X on screen at t?" → yes iff some frame's valid_from_ms <= t < valid_to_ms — no extra calls needed

  • audio-only job → returns the transcript slice plus a no-frames note (that is expected)

  • anti-example: whole-video summary → get_transcript(format="text"), not a chain of get_moments

  • anti-example: need more than 3 frames of a range → get_frames(start_ms=..., end_ms=..., max_frames=6)

  • keep ranges under ~30 s; a 5-min "moment" dilutes the bundle and wastes tokens

ParametersJSON Schema
NameRequiredDescriptionDefault
end_msYes
job_idYes
start_msYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds behavioral details beyond annotations: returns up to 3 frames, OCR, wall-clock range, audio-only handling, frame paths. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Very informative but lengthy. However, it is well-structured with clear sections and front-loaded key purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Complete for a tool with no output schema: covers return structure, edge cases (audio-only, diarized), and usage patterns.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema coverage, the description compensates fully, explaining start_ms/end_ms with examples, padding advice, and use cases.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Explicitly states it returns an evidence bundle with transcript slice, up to 3 frames, OCR, and time range. Distinguishes from siblings like get_transcript and search.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit when to use (one call per finding) and when not to use (broad exploration, keyword lookup). Includes multiple examples and anti-examples.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_transcriptA
Read-onlyIdempotent

Retrieve the transcript of a processed job, lazily and paginated. Formats: "segments" (default — seq, t_ms, t_wall when known, speaker when diarized, text), "text" (plain prose; "S1:" prefixes at speaker changes), "srt" (subtitles, speaker-prefixed cues). Diarized jobs also return the roster, attribution_precision, saved speaker_name values, raw OCR name_candidates, and bounded pending-review names plus old-roster context after a relabel. Pre-0.3.1 video jobs may return name_candidates_note because their flat OCR is readable but less useful for hints. Pending names are evidence to re-check, never active identities. A stale pending label can only be removed with label_speakers(..., labels={"Sx":null}). Raw S labels remain canonical. Responses are capped (~8k tokens): when truncated=true, continue from the returned next_start_ms. When NOT to use: to find one keyword (use search) or to inspect one moment with visuals (use get_moment). Examples:

  • get_transcript(job_id="a1b2c3d4e5f60718") — whole transcript of a short recording

  • get_transcript(job_id="a1b2c3d4e5f60718", start_ms=0, end_ms=120000) — just the first two minutes

  • get_transcript(job_id="...", format="text") — prose block for summarization

  • get_transcript(job_id="...", format="srt") — subtitle export the user asked for

  • diarized job: segments carry "speaker" + a roster header (top-12 by talk time; speakers_truncated counts the rest)

  • "what did S2 say?" → format="segments", collect entries with speaker=="S2" (labels are in order of first voice)

  • got truncated=true with next_start_ms=421500 → get_transcript(job_id="...", start_ms=421500)

  • user: "what was said between 5:00 and 6:30?" → start_ms=300000, end_ms=390000

  • legacy video name_candidates_note → explain the limitation; safe regeneration uses force=true+diarize=true

  • correlate speech with logs: each segment's t_wall lines up with your log timestamps

  • no speaker fields on a meeting job → re-run process_media with diarize=true (adds them without re-transcribing)

  • attribution_precision="segment" → force=true+diarize=true is required for exact word boundaries

  • pending context points to old evidence; stale labels accept null removal, never a new name

  • anti-example: "where did they mention checkout?" → search(job_id, "checkout"), not full paging

  • anti-example: screenshots around a remark → get_moment(job_id, start_ms, end_ms)

ParametersJSON Schema
NameRequiredDescriptionDefault
end_msNo
formatNosegments
job_idYes
start_msNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark readOnly, idempotent, non-destructive. The description substantially adds context: pagination truncation with next_start_ms, diarized roster details, speaker label semantics, stale-label removal, and legacy caveats. No contradiction with annotations; description enriches the safety profile with operational behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Though lengthy, it is dense and well-organized: core purpose first, then format spec, then pagination, then 'When NOT to use', then a long list of concrete examples with anti-examples. Every sentence adds value—no filler. Structure front-loads the most critical info and uses formatting (bullets) to aid scanning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has complex behavior (diarization, multiple formats, pagination, legacy notes). The description covers every aspect an agent needs to call it correctly: output shapes, pagination continuation, roster/attribution details, speaker label handling, and correlation with logs. No missing context even without an output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must fully compensate. It explains job_id implicitly, start_ms/end_ms for range, and format by listing the three enum values and their output structures. It goes beyond schema by describing what each format returns (segments fields, text prefixes, srt cues) and how to use start_ms for continuation. Complete semantic coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Retrieve'), resource ('transcript of a processed job'), and key behaviors (lazily, paginated). It explicitly differentiates from siblings by naming search and get_moment as alternatives for different intents, so an agent can select it correctly just from the description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Includes a dedicated 'When NOT to use' section naming alternatives (search, get_moment), plus extensive examples and anti-examples covering format selection, paging, diarized handling, and legacy cases. Guidance is explicit and conditions for choosing alternatives are crystal clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

label_speakersA
Idempotent

Persist VERIFIED human-readable names for anonymous S1/S2/… labels on one diarized job. labels is a patch: a 1-100 character name saves it; null or blank removes it. Optional evidence (max 500 characters per label) records why the mapping is trusted. Raw labels remain canonical in JSON; names appear separately and in text/SRT display. The write is atomic, locked, local, and idempotent. OCR name_candidates are raw hints only and are never saved automatically. A relabelled job can carry pending-review names with source-roster anchors. For a label still in the roster, an explicit patch confirms, replaces, or removes only that entry. For a stale pending label, only an explicit null removes its name, evidence, and context. When NOT to use: before diarization, or when a name is only a guess without human/screen evidence. Examples:

  • label_speakers(job_id="...", labels={"S1":"Vera"}) — save one verified mapping

  • label_speakers(job_id="...", labels={"S1":"Vera","S2":"Tom"}) — patch several labels

  • save evidence: label_speakers(job_id="...", labels={"S1":"Vera"}, evidence={"S1":"intro at 1200ms"})

  • screen name plate confirms S2 → save the name and cite its frame timestamp in evidence

  • label_speakers(job_id="...", labels={"S2":null}) — remove S2's name and evidence together

  • label_speakers(job_id="...", labels={"S2":" "}) — blank also removes the mapping

  • named S1, new proof → label_speakers(job_id="...", labels={}, evidence={"S1":"title card at 0ms"})

  • two Alexanders may map to S2 and S5 — duplicate names are allowed and search covers both

  • response roster carries speaker_name beside label; raw S labels are never replaced

  • name_candidates may be UI text or another person's name → inspect frames before deciding

  • unknown label or a name over 100 characters → error lists the valid roster labels

  • fresh session: get_transcript returns saved names; do not infer the mapping again

  • stale pending S3 → labels={"S3":null}; never assign a name while S3 is outside the roster

  • anti-example: uncertain identity → keep S anonymous until evidence verifies the name

ParametersJSON Schema
NameRequiredDescriptionDefault
job_idYes
labelsYes
evidenceNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses atomicity, locking, locality, and idempotency, plus null/blank removal semantics, duplicate-name allowance, and that name_candidates are never persisted automatically. This goes well beyond the annotations' idempotentHint and adds meaningful operational context without contradicting the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core contract and includes a dedicated 'When NOT to use' section, but the long example list contains some redundancy (e.g., null and blank removal are both demonstrated). Still, the length is largely warranted by the tool's edge-case-heavy behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers success behavior, error conditions, the response roster carrying speaker_name, interaction with stale labels, and the relationship to get_transcript and name_candidates. With the output schema present, nothing an agent needs to call this safely is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, but the description fully compensates: labels is defined as a patch where 1-100 character names save and null/blank removes, evidence is capped at 500 characters per label, and job_id refers to a diarized job. Concrete examples map each parameter to realistic calls.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence names a precise verb and resource: 'Persist VERIFIED human-readable names for anonymous S1/S2/... labels on one diarized job.' It also distinguishes itself from get_transcript by clarifying names are a separate layer from raw labels, and the 'When NOT to use' section reinforces scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly states when not to use the tool: before diarization or when the name is only a guess without human/screen evidence. It also points to get_transcript for reading already-saved names and warns against assigning names while a label is outside the roster.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_jobsA
Read-onlyIdempotent

List processed recordings, newest first: job_id, source filename, duration, created, wall-clock start, segment/frame counts. The store is content-addressed — the same file maps to the same job even after renames or moves, and jobs persist across sessions and machines restarts. When NOT to use: as a health check or before every call — job_ids are stable, remember them. Examples:

  • user: "triage the recording I processed this morning" → list_jobs() → pick by filename + created

  • user names neither job_id nor path → list_jobs() first; only ask if still ambiguous

  • resume yesterday's analysis in a fresh conversation → list_jobs() → reuse its job_id directly

  • file was renamed after processing → match by duration/created; the content hash ignores names

  • wall_clock.start answers "WHEN was this session?" — pick the job from "yesterday around 15:00"

  • after CLI batch pre-processing (talkthrough-mcp process big.mov) the job shows up here — query it

  • two jobs with the same filename → the newer created one is usually the re-recording

  • diarized jobs show "speakers": N — "the 4-person meeting from Tuesday" is findable at a glance

  • URL jobs carry "origin" (provider, provider_id, title) — "the YouTube video from yesterday" is findable

  • empty list → nothing processed on this machine yet; ask the user for a file path or a public URL

  • job disappeared → likely talkthrough-mcp gc cleaned it; re-run process_media on the file (same id)

  • anti-example: checking whether a NEW file is processed → just call process_media, it is idempotent+instant

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, and the description adds behavioral context: content-addressed store (same file maps to same job despite renames), persistence across restarts, empty list semantics, and gc cleanup behavior. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core purpose and output fields, then structured into 'When NOT to use' and examples. It is long but every segment adds decision value; a slight reduction in redundancy would make it tighter, yet nothing is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers ambiguity resolution (same filename, renamed files), temporal queries, CLI interplay, URL jobs, diarization, empty results, and job disappearance. With an output schema present and no input params, this is more than enough contextual guidance for reliable invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters and the input schema coverage is trivially complete at 100%. Per the rubric, a no-parameter tool earns a baseline 4; the description also usefully explains output field semantics (wall_clock.start, origin, speakers) that help an agent interpret rows.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'List processed recordings, newest first' and enumerates the exact fields returned (job_id, source filename, duration, created, wall-clock start, segment/frame counts). This is a specific verb+resource that clearly distinguishes list_jobs from siblings like process_media or get_transcript.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Contains a dedicated 'When NOT to use' section plus 12 examples covering triage, resume, renamed files, CLI pre-processing, duplicate filenames, and missing jobs. The anti-example explicitly routes new-file checks to process_media, leaving no ambiguity about when to call this tool vs alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

process_mediaA
Idempotent

Ingest a LOCAL video or audio file and make it queryable: validates the file, transcribes speech locally (whisper), extracts scene-change keyframes, OCRs on-screen text, resolves the wall-clock start time, and (opt-in) labels who said what via local speaker diarization. Returns a compact summary (job_id, media info, wall_clock, transcript preview, speaker roster when diarized) — full data stays on disk and is served lazily by the other tools. Idempotent by content hash: re-calling on an already-processed file returns instantly. For MULTI-PERSON recordings (meetings, interviews, calls) diarize=true is part of a proper analysis — pass it even when the user only asks for a summary. num_speakers is a target the clusterer may not reach, not a constraint — the payload says when a re-run changed nothing (labels_changed). If an amend changes the labels, verified names become pending-review evidence rather than active identities, with old-roster anchors for re-checking. Current pending labels can be confirmed/replaced/removed; stale labels can only be removed with an explicit null patch. Full force reprocessing of a job with saved or pending identities requires diarize=true and preserves every old identity as pending review against the rebuilt roster; without diarization it refuses before changing the stored job. When NOT to use: to re-fetch data you already processed (use the retrieval tools), or for URLs — local file paths only; a public video/audio URL goes to process_url. Examples:

  • process_media(path="/Users/sam/Desktop/bug-repro.mov") — narrated screencast, defaults are right

  • meetings: model="large-v3-turbo" + vocabulary=<attendees, terms> + num_speakers=N — turbo's extra cost is trivial

  • process_media(path="/tmp/standup.m4a") — audio-only: transcript tools work, frame tools will error

  • process_media(path="/rec/panel.mov", diarize=true, num_speakers=4) — headcount known? ALWAYS pass it: best accuracy

  • relabel amend → names become pending with old anchors; stale labels are removable only with null

  • error mentions [diarization] → run uvx --python ">=3.11,<3.14" "talkthrough-mcp[diarization]"

  • know the attendees? process_media(path=..., vocabulary="Anastasia, Evgenii, OKR") — names+jargon survive STT

  • user: "analyze/summarize this meeting" → include diarize=true — speaker structure is not optional extra credit

  • noisy threshold roster (clusters ≫ people)? ASK your user for the real headcount, then re-run with num_speakers=N

  • cap_hit or sampling_interval_s in summary → for slide hunts raise TALKTHROUGH_MAX_FRAMES or use extract_frame

  • summary shows wall_clock=null → ask when recording started, re-call with recorded_at=... and force=true

  • transcript garbled or language_probability low → re-call with model="large-v3-turbo" (or language="ru") + force=true

  • after success, do NOT dump everything — continue with get_transcript / get_moment / search on the job_id

  • anti-example: frames from an already-processed job → get_frames(job_id=...), never process_media again

  • named job + force=true → include diarize=true; old identities return as pending review, never silently vanish

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYes
forceNo
modelNo
diarizeNo
languageNo
vocabularyNo
recorded_atNo
num_speakersNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description goes far beyond the annotations: it discloses idempotency by content hash, lazy serving from disk, refusal to re-process saved identities without diarize=true, pending-review label behavior, null-patch removal for stale labels, and failure signals like wall_clock=null or cap_hit. These behavioral traits are not visible in the annotations and are critical for correct invocation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with a dense summary, then organized into not-to-use guidance, examples, and troubleshooting. It is long, but the length is mostly earned given 0% schema coverage and the tool's complex diarization/identity edge cases; a slightly tighter arrangement would improve conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex ingestion tool with 8 parameters, no schema descriptions, and nuanced behavioral rules, the description is remarkably complete. It covers inputs, outputs, failure modes, parameter semantics, follow-up actions, environment setup for diarization, and explicit when-not-to-use cases. An agent has everything needed to call this tool correctly in a wide range of scenarios.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description carries the full burden, and it delivers. It adds real meaning to path (local only), diarize (required for multi-person), num_speakers (a target, not a constraint), model and language (when to upgrade/re-run), vocabulary (names+jargon survive STT), recorded_at (needed when wall_clock=null), and force (required alongside diarize for named jobs). Every parameter is given operational context beyond its bare schema type.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: ingest a LOCAL video or audio file and make it queryable, then enumerates the concrete pipeline (validate, transcribe, keyframes, OCR, wall-clock, diarization). It clearly differentiates itself from siblings like process_url and the retrieval tools, so an agent can select it without ambiguity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly states when NOT to use the tool: re-fetching processed data should use retrieval tools, and URLs belong to process_url. Numerous examples map user intents (meeting summaries, audio-only files, known headcounts, re-processing) to the correct parameters, including an anti-example that warns against re-processing already-processed jobs.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

process_urlA

Download ONE public video/audio URL once (this is the only tool that uses the network), then run the same LOCAL pipeline as process_media: transcript, keyframes, OCR, wall-clock, optional diarization. Supported: direct https:// links to a media file (mp4/mov/webm/mkv/ogv/m4a/mp3/wav/ogg/flac), one public YouTube video (watch, youtu.be, shorts, a completed live), and any public video PAGE yt-dlp can read — Instagram (public reels/posts), TikTok, Wikimedia Commons, pages with an HTML5/HLS player, other sites as far as their yt-dlp extractor works anonymously (Vimeo does not). Not supported: playlists, channels, active live streams, private/members-only/age-restricted/DRM videos, cookies/logins; sites that hide a video behind a login or a bot wall fail with a clear reason. The downloaded source is kept inside the job, so extract_frame works later without network; the raw URL is never stored (only a hash, the provider id/host and a bounded title). A repeat call on the same URL serves the stored job without touching the network unless refresh=true. Job ids stay content hashes: the same video from two URLs is one job. YouTube and other pages need the optional [url] extra. The provider's upload date is NOT the recording start: wall_clock stays null unless recorded_at is passed. When NOT to use: for local files (process_media), or to re-fetch data you already processed (use the retrieval tools). Examples:

  • process_url(url="https://youtu.be/nHfGfEiVdE8") — one public YouTube video, defaults are right

  • process_url(url="https://www.youtube.com/watch?v=ID&list=PL...") — the playlist part is ignored: ONE video

  • process_url(url="https://cdn.example.com/recordings/standup.mp4") — direct https link to a media file

  • process_url(url="https://www.tiktok.com/@nasa/video/7…") — a public video page; origin.provider names the site

  • meeting from a link: process_url(url=..., diarize=true, num_speakers=3, vocabulary="Vera, Tom, OKR")

  • non-English narration: process_url(url=..., model="large-v3-turbo", language="ru")

  • known recording start: process_url(url=..., recorded_at="2026-09-05T14:00:00+02:00") — enables t_wall

  • the video changed on the provider → process_url(url=..., refresh=true): new download, maybe a new job_id

  • re-anchor or change the model on a stored URL job → process_url(url=..., recorded_at=..., force=true), no download

  • error mentions [url] → run uvx --python ">=3.11,<3.14" "talkthrough-mcp[diarization,url]" and restart

  • playlist / channel / live / private URL → clear error; pass a single public video URL instead

  • "bot check"/"sign-in" refusal on Instagram/TikTok → the site blocked anonymous access; report it, no workaround

  • origin.published_at is the provider's upload time, not when the recording was made — never use it as t_wall

  • after success continue with get_transcript / search / get_moment on the job_id — never re-download

  • anti-example: a file on disk → process_media(path=...); process_url is only for https URLs

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
forceNo
modelNo
diarizeNo
refreshNo
languageNo
vocabularyNo
recorded_atNo
num_speakersNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover read-only/destructive hints, but the description adds substantial non-obvious behavior: it is the only network-using tool, downloaded sources are retained for later extract_frame, raw URLs are never stored, job ids are content hashes, repeated calls are cached unless refresh=true, and wall_clock stays null unless recorded_at is provided. No contradiction with annotations exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but well structured into core definition, supported/unsupported inputs, behavioral caveats, when-not-to-use, and topical examples. Each section provides distinct operational information; the minor repetition about upload-date versus recording-start is acceptable given the risk of misuse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a network-capable tool with 9 parameters, no parameter descriptions, and an output schema present, the description is exceptionally complete. It covers invocation semantics, failure modes, caching/deduplication, privacy behavior, provider limitations, and follow-up workflow with retrieval tools. Nothing essential for correct use appears missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must carry the semantic burden. It does so via examples covering all nine parameters: url, refresh, force, model, language, diarize, num_speakers, vocabulary, and recorded_at. It also explains important parameter interactions, such as force avoiding re-download and playlist parts being ignored in url.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: download exactly one public video/audio URL and run the same local pipeline as process_media. It clearly distinguishes itself from local-file processing and retrieval tools, and enumerates supported and unsupported URL types, making the tool's scope unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description includes an explicit 'When NOT to use' section naming process_media and the retrieval tools, plus an anti-example for local files. The extensive examples also explain when to use refresh, force, diarize, model, language, vocabulary, and recorded_at, giving an agent concrete decision guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 1 tool updatev0.4.0
    • Addedprocess_url
  2. 2 tool updatesv0.3.0
    • Addedlabel_speakers
    • Changedsearch1 field changed
      • addedInput schema / properties / match_mode
        Added value: +{
+  "default": "all_words",
+  "enum": [
+    "all_words",
+    "any_word"
+  ],
+  "title": "Match Mode",
+  "type": "string"
+}
  3. 6 tool updatesv0.2.4
    • Addedextract_frame
    • Addedget_frames
    • Addedget_transcript
    • Addedlist_jobs
    • Addedprocess_media
    • Addedsearch
  4. 3 tool updatesv0.2.3
    • Removedextract_frame
    • Removedlist_jobs
    • Removedsearch
  5. 3 tool updatesv0.2.0
    • Removedget_frames
    • Removedget_transcript
    • Removedprocess_media
  6. 7 tool updatesv0.1.0
    • First observedextract_frame
    • First observedget_frames
    • First observedget_moment
    • First observedget_transcript
    • First observedlist_jobs
    • First observedprocess_media
    • First observedsearch
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.8/5.0

Scored across 9 tools

Disambiguation5/5

Each tool targets a distinct resource/action: local vs URL ingestion, transcript retrieval vs keyword search, stored keyframes vs exact re-extraction vs combined evidence bundles. The descriptions include explicit anti-examples that reinforce the boundaries, making misselection unlikely.

Naming Consistency4/5

Eight of nine tools follow a clear verb_noun pattern: process_media, process_url, get_transcript, get_frames, get_moment, list_jobs, label_speakers, extract_frame. The single bare verb 'search' is a minor deviation, but the overall naming remains predictable and readable.

Tool Count5/5

Nine tools is well-scoped for a media-analysis server. Each tool maps to a distinct phase of the workflow — ingestion, listing, retrieval, searching, evidence gathering, and speaker identity management — with no redundant additions.

Completeness4/5

The ingest/read/update side is well covered: process_media and process_url create jobs, list_jobs/get_transcript/get_frames/get_moment/search read them, and label_speakers plus force reprocessing update them. There is no MCP-level job deletion tool (gc is external/CLI-only) and no standalone OCR-only dump, which are minor workaround-level gaps rather than core analysis failures.

Maintenance

ActivityActive
ResponsivenessWithin a week

Resources

Related MCP Connectors

Related MCP Servers