talkthrough-mcp

Quickstart · Tools · FAQ · Troubleshooting · Changelog · Contributing

Feedback ingestion for AI agents. Record your screen and talk; your agent does the rest — files the bugs, writes the spec, builds the backlog.

talkthrough-mcp is a local-first MCP server that turns a narrated screen recording (or any video/audio file) into agent-ready structured data: timestamped transcript segments, scene-change keyframes, OCR'd on-screen text, and wall-clock anchoring. Everything is served through lazy retrieval tools, so a 30-minute recording never floods the model context — the agent pulls exactly the transcript slice, moment bundle, or frame it needs.

There is no LLM inside the server and no cloud anywhere in the path: ffmpeg, faster-whisper, and RapidOCR run on your machine, and the calling agent brings the intelligence. What makes it different from screen-recorder SaaS and video-analyzer MCPs: it works on arbitrary local files, it ships the agent workflows (server prompts + example agents), and it anchors every timestamp to wall-clock time — so "the moment I said the checkout hung" maps straight to the right window of your server logs.

Quickstart

One command, no system dependencies: ffmpeg falls back to a bundled build, OCR is pip-only, and whisper models download themselves on first use. The only prerequisite is uv (brew install uv or curl -LsSf https://astral.sh/uv/install.sh | sh).

Claude Code

Two install paths — pick one, not both (the plugin already includes the server; installing both would register it twice):

Server only — the 7 tools + 5 prompts, and nothing else on your system. Choose this for a minimal setup, or when you manage MCP servers yourself across several clients:

claude mcp add -s user talkthrough -- uvx talkthrough-mcp

Full plugin — the same server, plus native slash commands (/talkthrough:triage-recording, …) that handle the ceremony for you, a ready-made triage subagent, and an agent skill that teaches Claude the workflow. Choose this for the best out-of-the-box experience:

/plugin marketplace add korovin-aa97/talkthrough-mcp
/plugin install talkthrough@talkthrough

Every other MCP client

claude_desktop_config.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/claude-desktop/

~/.cursor/mcp.json (or project .cursor/mcp.json):

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/cursor/

~/.codex/config.toml (or project-scoped .codex/config.toml in trusted projects):

[mcp_servers.talkthrough]
command = "uvx"
args = ["talkthrough-mcp"]

More: integrations/codex/

~/.gemini/settings.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/gemini-cli/

cline_mcp_settings.json (via MCP Servers UI):

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/cline/

~/.openclaw/openclaw.json:

{
  "mcp": {
    "servers": {
      "talkthrough": {
        "command": "uvx",
        "args": [
          "talkthrough-mcp"
        ]
      }
    }
  }
}

More: integrations/openclaw/

opencode.json (project) or ~/.config/opencode/opencode.json:

{
  "mcp": {
    "talkthrough": {
      "type": "local",
      "command": [
        "uvx",
        "talkthrough-mcp"
      ],
      "enabled": true
    }
  }
}

More: integrations/opencode/

~/.config/goose/config.yaml:

extensions:
  talkthrough:
    enabled: true
    type: stdio
    cmd: uvx
    args: ["talkthrough-mcp"]

More: integrations/goose/

~/.copilot/mcp-config.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/copilot-cli/

~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/windsurf/

settings.json (Zed):

{
  "context_servers": {
    "talkthrough": {
      "source": "custom",
      "command": {
        "path": "uvx",
        "args": [
          "talkthrough-mcp"
        ]
      }
    }
  }
}

More: integrations/zed/

Any other MCP stdio client uses the same server command: uvx talkthrough-mcp. Per-engine folders with exactly these snippets plus verification steps live in integrations/; agents can self-install via llms-install.md.

Local checkout (development)

git clone https://github.com/korovin-aa97/talkthrough-mcp
claude mcp add talkthrough -- uv run --directory /path/to/talkthrough-mcp talkthrough-mcp

Then, in your agent:

Process ~/Desktop/recording.mov and triage it — or just invoke the triage-recording server prompt.

Related MCP server: mcp-feedback-enhanced-gw

Tools

Every tool description ships 10+ usage examples, so agents pick the right tool without extra prompting.

Server prompts (slash commands in MCP clients)

The same prompts live as plain files in examples/prompts/ if your client doesn't surface MCP prompts. The findings contract used by triage-recording is examples/output-contract.schema.json.

Works as a skill too (no MCP required)

The same workflow ships as a cross-engine Agent Skill at .agents/skills/talkthrough/ — Claude Code, Codex CLI ($talkthrough), Cursor, Copilot, Gemini CLI, Goose and other SKILL.md-compatible tools read it. Agents without MCP wiring can drive the CLI directly: talkthrough-mcp process recording.mov --json prints the same summary the MCP tool returns, and the job store is shared either way.

Wall-clock anchoring

Every timestamped result carries both t_ms (video-relative) and t_wall (ISO 8601 real time) once the recording start is known. Resolution ladder:

1. recorded_at parameter (agent/user override) → confidence exact

2. QuickTime com.apple.quicktime.creationdate tag, carries the local timezone (QuickTime Player recordings; ⌘⇧5 wrote it before macOS 26) → high

3. Container creation_time tag (UTC) → medium — macOS 26+ ⌘⇧5/ReplayKit screen recordings land here (no creationdate tag anymore); pass recorded_at= when local-tz t_wall matters

4. File mtime minus duration (recorders finalize files at recording END) → low

5. Nothing → tools still work with relative t_ms only

Why it matters: "the upload spinner froze here" becomes a ±30 s grep window in your server logs.

Privacy

Everything runs locally: your recordings never leave your machine, speech is transcribed by a local whisper model, OCR is local ONNX inference, and there is no telemetry. The only network access is one-time tool/model downloads (ffmpeg build, whisper model, OCR models).

Languages

Narration in any of Whisper's ~99 languages works: the language is auto-detected per recording, and the summary reports both language and language_probability so agents can tell a confident detection from a shaky one (silence or music at the start can fool the detector — pin it with language="ru" and force=true when that happens).

Pick the model for your languages — per call (model= parameter, agents do this themselves when a transcript comes back garbled) or as the server default (TALKTHROUGH_WHISPER_MODEL):

Tips that work in every language: pass product names via vocabulary="Term1, Term2" (biases the decoder so jargon survives), and note that the workflow prompts instruct agents to write digests in the narrator's language while keeping quotes verbatim — the server never translates (exact quotes are evidence; translation is the agent's job).

On-screen text (OCR) defaults to RapidOCR's Latin + Chinese models. For other scripts set TALKTHROUGH_OCR_LANG to your language — ru/uk (→ the eslav pack), ja, ko, ar, hi, el, th, or any RapidOCR pack name like cyrillic — and reprocess with force=true; the matching recognition model downloads once. Spoken-language support is unaffected either way.

Configuration

CLI

The pipeline is also a CLI — useful for pre-processing long recordings outside an agent session (the store is content-addressed, so the agent then queries the same job instantly):

talkthrough-mcp process ~/Videos/long-session.mov   # prints the summary
talkthrough-mcp process demo.mov --json             # machine-readable
talkthrough-mcp gc --keep-days 30                   # clean the job store
talkthrough-mcp serve                               # stdio MCP server (default)

First run notes: missing system ffmpeg triggers a one-time static-ffmpeg download; the first transcription downloads the whisper model (~460 MB for small); both are cached. After that, expect roughly 3× faster than real time on an Apple-Silicon CPU with the default model, OCR included (a 2-minute clip processes in ~40 s) — and instant re-runs on the same file. Progress streams as MCP progress notifications, and the CLI prints stage lines. More: docs/TROUBLESHOOTING.md.

Windows (best-effort)

CI runs lint, the unit suite, and a full CLI smoke on windows-latest (static-ffmpeg Windows build, whisper tiny transcription, OCR, and the instant idempotent re-run). Notes: the per-job lock is POSIX fcntl and degrades to a no-op on Windows — fine for a single-user machine; quote paths with spaces (uv run talkthrough-mcp process "C:\Videos\Screen Recording.mp4"). Windows is not a release gate — if something breaks, please open an issue.

Supported inputs

Video: .mov .mp4 .webm .mkv — audio-only: .m4a .mp3 .wav .ogg .flac (transcript tools only; frame tools explain why they're unavailable). Local files only.

Limitations

Honest edges, so you can decide fast:

• One speaker stream. No diarization yet — "who said it" isn't tracked (#4).

• Local files only. No URL/YouTube ingestion (#5) — download first.

• Keyframes + transcript, not motion analysis. A glitch between scene changes can be invisible in the frame set; extract_frame re-checks any instant, but frame-by-frame motion reasoning is your multimodal model's job.

• STT quality tracks the model you pick. The default small favors speed; non-English narration wants model="large-v3-turbo" (see Languages).

• OCR reads crisp UI text well; tiny or low-contrast print is best-effort.

• Wall-clock confidence depends on recorder metadata — worst case pass recorded_at= (see the ladder above).

• Windows is best-effort (see above).

How it compares

FAQ

Why not just upload the video to a multimodal model (e.g. Gemini)? For a short, non-sensitive clip — do that. The trade-offs appear with length and sensitivity: an hour of screen recording costs on the order of a million tokens per question, the file leaves your machine, and you still can't map a remark to 14:32:07 UTC to grep your server logs. talkthrough indexes once, locally, then answers any number of follow-ups from the index.

Why not screenpipe? Different job. screenpipe is an always-on recorder of your machine going forward (commercial license). It can't open the .mov a teammate or customer just sent you. talkthrough analyzes any file it's handed — the two compose fine.

There are agent skills that "watch" videos. Why a server with an index? Watch-style skills push a budgeted frame dump into the context window (and go sparse on long videos), often call cloud STT for the audio, and keep nothing. talkthrough builds a persistent local index — transcript + OCR, full-text searchable — retrieves exact frames lazily, anchors everything to wall-clock time, and answers the next question without reprocessing.

I use Jam for bug reports — do I need this? Keep Jam for browser bugs: console+network captured at record time is great evidence. talkthrough covers what a browser extension can't — desktop apps, mobile screencasts, ops incidents, meetings, any file — with no account, and correlates with server-side logs via wall-clock time.

Can't I just script ffmpeg + whisper myself? Yes — that's exactly this pipeline. What you'd be rebuilding: scene-change detection with perceptual dedup, OCR, transcript+OCR search, the wall-clock ladder, MCP tools with embedded usage examples, five workflow prompts, and a findings contract. One uvx command instead of an afternoon of glue.

Is it really local? What leaves my machine? Nothing at runtime. The network is used only for one-time downloads (ffmpeg build, whisper/OCR models). No telemetry. See Privacy — and SECURITY.md treats a violation of this promise as a vulnerability.

For agents & tooling

Machine-readable entry points, so AI agents can install and use this server without a human reading docs:

• llms-install.md — step-by-step install instructions for agents

• llms.txt — index of the documentation

• .agents/skills/talkthrough/SKILL.md — an Agent Skill teaching the tool workflow; discovered automatically inside a checkout by Codex CLI ($talkthrough) and readable by Claude Code, Cursor, Copilot, Gemini CLI and other SKILL.md-compatible tools

• AGENTS.md — instructions for coding agents contributing to this repo

• server.json — MCP registry manifest

• integrations/ — per-engine adapters, all generated from one source of truth and drift-tested (incl. the Claude Code plugin under integrations/claude-code/)

Roadmap (not in v1)

URL/YouTube ingestion · speaker diarization · cloud STT · embeddings/semantic search · hosted/remote mode · .mcpb bundle · whisper.cpp backend

License

MIT