Skip to main content
Glama

Framesleuth

Local video → structured context for coding agents, exposed over MCP.

Feed Framesleuth a video and it reads it frame by frame, folds in optional browser sidecars, and produces a structured Context Bundle. Any video works: a bug recording, a feature demo, a design walkthrough, a Loom, a phone capture.

The bundle is served over MCP, so a VS Code agent, another coding agent, or your own system can drive the analysis and use the result to fix a bug, change a feature, or build something new, grounded in what the video actually shows.

Capture happens outside this repo, which holds the analysis agent only. A browser capture extension can record a session and post the video plus sidecars to the local API.

Everything runs locally. Nothing leaves your machine.

Quick start

Going from a video to a grounded change inside VS Code? See Use with VS Code & Claude (MCP): connect the bundled MCP server, then turn a recording into a fix, a feature, or a new build.

Fastest: one command with Docker

One command brings up the model server, the models, and the API. No Python, no virtualenv, no manual model setup.

git clone https://github.com/thestackhub1/framesleuth-agent.git
cd framesleuth-agent
docker compose up            # or: ./scripts/dev_up.sh

Compose picks up docker-compose.override.yml automatically; that file adds the Ollama server, the model-pull job, and the model volume. The first run pulls the vision and coder models (qwen2.5vl and qwen2.5-coder:7b, ~11 GB total) into a Docker volume, then starts the backend on http://127.0.0.1:8010. Later runs are instant. It's ready when the health check says healthy:

curl -s http://127.0.0.1:8010/v1/healthz | python -m json.tool   # "status": "healthy"

That's the whole setup. Run your first analysis below, or connect the MCP server in your editor (VS Code & Claude).

docker compose logs -f                  # follow progress / model download
docker compose down --remove-orphans    # stop  (add -v to also delete model volumes)

The stack runs its own Ollama on the internal Docker network and never publishes its port, so it won't clash with a native Ollama on :11434. The only host port is the API on :8010.

Already running Ollama natively with the models pulled? The Docker stack ships its own Ollama and would download them again. Use the direct path below instead. It reuses your existing Ollama and is faster, especially on macOS, where Docker can't reach the GPU.

On macOS, or anywhere without a GPU, Docker runs the models on CPU and the vision model is slow. On Linux with an NVIDIA GPU, uncomment the deploy: block on the ollama service in docker-compose.override.yml.

To run only the backend container against a native or external model server, use docker compose -f docker-compose.yml up. The base compose file defaults to native Ollama on http://host.docker.internal:11434; override VLM_URL and CODER_URL for another server.

Docker users: don't cp .env.example .env. If you already did, comment out VLM_URL and CODER_URL in it. Compose reads .env and those values beat the defaults above, and .env.example ships the native 127.0.0.1, which inside a container means the container itself. The symptom is a backend that starts cleanly and then can't reach any model.

Run your first analysis (curl)

Once the API reports healthy, either setup path, three calls take you from a video to a Context Bundle. Analysis is async: submit, poll, read.

No recording handy? Generate a throwaway one. It exercises the whole pipeline and takes about a second.

uv run python scripts/make_sample_video.py     # writes sample.mp4
# 1. Submit any screen recording (mp4/webm). Returns 202 { job_id, ... }
JOB=$(curl -s -F "video=@sample.mp4" http://127.0.0.1:8010/v1/analyze \
  | python -c "import sys, json; print(json.load(sys.stdin)['job_id'])")

# 2. Poll until state is "done" (queued → running → done)
curl -s "http://127.0.0.1:8010/v1/jobs/$JOB" | python -m json.tool

# 3. Read the Context Bundle
curl -s "http://127.0.0.1:8010/v1/report/$JOB" | python -m json.tool

Step 1 takes optional form fields: -F intent="why does save hang?", -F skill=bug_report, -F action=fix. GET /v1/skills and /v1/actions list the choices. Prefer a UI? The Postman collection chains these calls for you.

Run it directly (no Docker — fastest on macOS, best for development)

You need Python 3.11+, uv, 8 GB+ RAM, and a local model server. ffmpeg isn't required, since PyAV bundles its own; if ffprobe happens to be on PATH it's used to detect an audio stream.

git clone https://github.com/thestackhub1/framesleuth-agent.git
cd framesleuth-agent

# 1. Models — native Ollama (uses the Mac GPU) is the quick path
ollama serve &                                  # skip if already running
ollama pull qwen2.5vl && ollama pull qwen2.5-coder:7b

# 2. Install — from uv.lock, so you get the versions CI actually tested
uv sync --frozen --extra dev
source .venv/bin/activate
python scripts/download_models.py               # optional: pre-warm ASR + check servers

# 3. Configure + start the API (binds 127.0.0.1:8010)
cp .env.example .env                            # already defaults to the Ollama path above
framesleuth-api                                 # or: uvicorn framesleuth.service.api:app --port 8010

# 4. Verify  (says so either way — a silent command is not a passing check)
curl -s http://127.0.0.1:11434/v1/models | grep -q qwen2.5vl \
  && echo "VLM ready" || echo "VLM NOT ready — run: ollama pull qwen2.5vl"
curl -s http://127.0.0.1:8010/v1/healthz | python -m json.tool   # status: healthy, vlm: ready

When /v1/healthz shows vlm: ready, recordings get a real classification (analysis_quality.level of full or partial). ready means the server answered and listed your VLM_MODEL. If the model was never pulled you get vlm: degraded with model '<name>' not loaded instead.

With no vision model reachable at all, Framesleuth degrades gracefully. It still produces a valid Context Bundle from the browser sidecars (console errors, failed requests, clicks) and records what was thin in analysis_quality. Narrate while you record and the audio transcript (asr) stage contributes too.

Something not working? Run the setup doctor. It runs under a plain python3 even when your virtualenv is broken, and prints a one-line fix for each problem: a stale or missing venv, framesleuth-api not on PATH, ffmpeg and render prerequisites, an unreachable backend or model server, a wrong VLM_URL.

python3 scripts/doctor.py

The common one: command not found: framesleuth-api, or a uv pip install error about a missing interpreter, means your active venv was deleted or moved. Fix it from the framesleuth directory: deactivate; unset VIRTUAL_ENV; uv sync --frozen --extra dev; source .venv/bin/activate.

Stop

# Stop the backend: Ctrl+C in its terminal, or
pkill -f framesleuth-api

# Stop Ollama (optional — leaving it running keeps the model warm)
pkill -f "ollama serve"              # macOS app users: quit Ollama from the menu bar

Related MCP server: Framesleuth

Architecture

Any video (mp4/webm) + optional sidecars
    ↓
Local Analysis Service (pipeline)
    ├─ Preprocess (PyAV: duration/fps/dims)
    ├─ Transcript (faster-whisper)
    ├─ Keyframes (visual-delta change scoring)
    ├─ Understanding (local vision model — Qwen2.5-VL by default)
    ├─ Fusion + Classification
    ├─ Extraction → Context Bundle
    ├─ Summarize (skill/system-prompt-driven)
    └─ Grounding (workspace search)
    ↓
Context Bundle
    ↓
MCP server + local HTTP API
    └─ consumed by any MCP client (VS Code agent, other agents, capture extension)

Features

  • Frame-by-frame understanding with a local vision model (Qwen2.5-VL by default; engine-agnostic)

  • Adaptive keyframe selection. Coverage-binned and visual-salience-ranked (AKS-style), with a build-aware budget for feature and design videos. Perceptual-hash dedup drops near-identical frames so the VLM budget goes on distinct content.

  • Bug and build. A feature class plus a structured build context: screens, UI components, a screen-to-screen user flow, design notes, and where to implement. An agent can build from it, not only diagnose.

  • Error detection and extraction from console, OCR, and UI state

  • Corpus-aware grounding. Error symbols or feature/UI nouns resolve to ranked file:line hits. Definitions are preferred, distinctive symbols weighted via IDF plus whole-word match, .gitignore respected, and the search bounded for large repos.

  • Trust signals. Per-field confidence, where agreeing signals across modalities corroborate each other, plus a task-aware actionability (ready/thin/insufficient) alongside the pipeline quality level.

  • Redaction-first design. Secrets (passwords, tokens, keys) and PII (emails, Luhn-valid card numbers, SSNs/phones, cloud keys) are scrubbed from OCR, captions, the transcript, and the raw sidecar streams before any of it reaches a model or is persisted. That covers the bundle and the sibling timeline.json, sidecars.json and transcript.json.

  • Observability. Per-stage timings land on every bundle (stage_timings) and live on GET /v1/jobs/{id}, so you can see where analysis time went.

  • Job lifecycle and delivery. Cooperative cancellation (DELETE /v1/jobs/{id}, checked between frames), a hard per-job timeout (JOB_TIMEOUT_S), crash recovery that fails orphaned jobs on restart rather than leaving zombies, SSE progress with explicit terminal events (GET /v1/jobs/{id}/events), a completion webhook (WEBHOOK_URL), real queue depth in /healthz, and TTL retention cleanup (BUNDLE_TTL_DAYS) swept at startup and periodically (RETENTION_SWEEP_INTERVAL_S).

  • Interaction overlay. A click/cursor sidecar with coordinates draws a marker on the matching keyframe, so the model sees where the user acted.

  • Cleaner transcripts. faster-whisper voice-activity filtering (ASR_VAD_FILTER) drops silence before decoding; the detected or forced language is recorded.

  • OCR backstop (optional ocr extra). A sparse VLM OCR on an error frame gets a second, independent Tesseract reading. Without the extra it's a no-op.

  • No data leaves your machine. Fully local, no telemetry, no cloud APIs.

  • Engine-agnostic. Swap Ollama, llama.cpp, or vLLM via config only.

  • Works on any video, not just bug recordings. A demo, a walkthrough, a talk, a phone clip: each yields a faithful summary and a timeline of key moments (summary, key_moments[]) rather than something forced into a bug shape. The bug-only fields (severity, expected/actual, repro steps) stay null instead of carrying fabricated placeholders.

  • Structured output. A canonical Context Bundle with evidence citations.

  • Configurable response. Pick a summary skill and an action mode (fix/implement/design/summarize/explain/triage/test/report/reproduce, auto-picked from the classification), plus a machine-readable suggested_actions menu and on-demand artifact renderers (markdown, GitHub issue, test plan).

  • Eval harness. Model-free classification, grounding, citation and faithfulness suites (python scripts/eval_harness.py --behavioral) run in CI on every push and PR: a GitHub Actions 3.11/3.12 matrix of ruff, black, mypy --strict, pytest behind a coverage gate, the eval harness against per-metric thresholds in evals/, and an OpenAPI-freshness check, plus a separate security job running pip-audit and pre-commit. The faithfulness suite proves every emitted key moment and step cites real, resolvable evidence.

  • Resilient. Handles no-audio videos, weak local models, and low-confidence cases.

  • HTML → video (frame-by-frame). Turn a self-contained HTML animation (CSS/JS/canvas) into MP4, GIF, or WebM via the render_html_video MCP tool or POST /v1/render-html. Frames are captured one at a time under a paused virtual clock and encoded to a color-correct H.264 MP4 (yuv420p+bt709, near-lossless): full color, no dropped frames, no quality loss, up to 4K and 5–60 fps. The Docker image includes it by default (headless Chromium + ffmpeg). On the direct path, add the render extra (see below); without it the endpoint returns 503 with an actionable message.

Enable & troubleshoot HTML → video

On Docker (docker compose up) this already works; the image bakes in Playwright, Chromium and ffmpeg. Build with --build-arg INSTALL_RENDER=false for a slimmer image without it. The steps below are for the direct path.

Playwright lives in an optional [render] extra rather than core, because it pulls a ~150 MB headless-Chromium browser the video→bundle pipeline never needs. (av, opencv and faster-whisper are core.) Install the extra and you're done. The Chromium build downloads on your first render, so there's no separate playwright install chromium step:

# In the same environment the server runs in:
uv sync --frozen --extra dev --extra render   # or --all-extras
# ffmpeg must be on PATH (brew install ffmpeg / apt-get install ffmpeg)

# Restart framesleuth-api, then verify (Chromium fetches itself on first render):
curl -s http://127.0.0.1:8010/v1/healthz | python -m json.tool
# → "render": {"playwright": true, "chromium": <true after first render>, "ffmpeg": true}

Set FRAMESLEUTH_AUTO_INSTALL_BROWSER=0 to disable the auto-download and run playwright install chromium yourself, e.g. in a locked-down environment.

The other optional extra is ocr. For the dedicated OCR backstop on error frames, run uv sync --frozen --extra dev --extra ocr and put the tesseract binary on PATH (brew install tesseract / apt-get install tesseract-ocr). Absent, it's a no-op: the VLM still does OCR, and the backstop only adds a second reading. Use ".[all]" for dev + render + ocr.

If render.ready is false, ask /v1/version for the details. /v1/healthz is public, so it omits the hint and python fields rather than publish the server's filesystem layout to an unauthenticated caller:

curl -s http://127.0.0.1:8010/v1/version | python -m json.tool
# → "render": {"ready": false, "hint": "...", "python": "/path/to/the/interpreter", ...}
# With API_TOKEN set, this endpoint is token-gated:
#   curl -s -H "Authorization: Bearer $API_TOKEN" http://127.0.0.1:8010/v1/version

render.hint tells you what's missing. When you followed the steps and still get "Playwright is not installed", it's usually one of two things: framesleuth-api is running from a different environment than the one you installed into (render.python names the interpreter it uses), or the server wasn't restarted.

Project structure

framesleuth/
├── framesleuth/              # Main package
│   ├── config.py            # Typed config (pydantic-settings)
│   ├── schemas.py           # Data contracts (Context Bundle, enums)
│   ├── errors.py            # Exception taxonomy
│   ├── logging_config.py    # Structured JSON logging, job-id correlation
│   ├── prompts.py           # VLM / classify / summary / fix prompt templates
│   ├── skills.py            # Built-in summary skills (summary, bug_report, ...)
│   ├── actions.py           # Action modes (fix/explain/triage/...) + suggested-actions menu
│   ├── render.py            # Artifact renderers (markdown / GitHub issue / test plan)
│   ├── clients/             # VLM, coder HTTP clients (OpenAI-compatible)
│   ├── pipeline/            # preprocess, asr, scenes, understand, fusion, classify,
│   │                        #   bug_extract, build_context, confidence, dedup, overlay,
│   │                        #   ocr, redact, summarize, sidecars, grounding, gif,
│   │                        #   atomic, html_render
│   ├── eval/                # harness.py — model-free behavioral suites
│   ├── orchestrator/        # graph.py — linear async stage pipeline
│   ├── jobs/                # store.py — SQLite job state + bundle index
│   ├── service/             # FastAPI HTTP endpoints
│   └── mcp_server/          # framesleuth MCP server (VS Code + any MCP client)
├── tests/                   # pytest tests + fixtures
├── scripts/                 # doctor.py (setup check), download_models.py, dev_up.sh,
│                            #   eval_harness.py, export_openapi.py
├── evals/                   # thresholds.json + baseline.json (the CI quality gate)
├── openapi.json             # generated API schema — the contract clients build from
├── postman/                 # HTTP API collection + environment
├── docs/                    # capabilities, use-with-vscode-and-claude, web-integration
└── pyproject.toml           # Dependencies and tool config

Development

Run tests

pytest tests/ -q                                        # fast: no coverage gate
pytest tests/ -q --cov=framesleuth --cov-fail-under=75  # what CI enforces

Regenerate the API schema (after changing any route)

python scripts/export_openapi.py --out openapi.json

CI fails if this file is stale; the website generates its typed client from it.

Run the eval gates

python scripts/eval_harness.py --behavioral   # see evals/README.md

Code quality

ruff check framesleuth tests
black --check framesleuth tests
mypy --strict framesleuth

Set up pre-commit hooks

pre-commit install

Docs, a short and focused set:

License

Apache-2.0


Capture client

Bug capture lives outside this repo. Any screen recording works, so you can drive the agent with your own video file. A browser capture extension can also record a session, collect browser sidecars (console errors, failed requests, clicks), and post the video plus sidecars to this agent's local API.

CORS is an exact allowlist. The local dev origins http://localhost:3000 and http://127.0.0.1:3000 are on by default; set ALLOW_LOCAL_DEV_ORIGINS=false on a hardened deployment to drop them. chrome-extension:// origins come from the IDs you list in CHROME_EXTENSION_IDS, empty by default, so a capture extension has to add its own. Everything else goes in WEB_ORIGINS, also empty by default: no remote site is trusted, framesleuth.com included. The agent answers Chrome's Private Network Access preflight, so an allowed origin can drive a backend running locally.

To let the hosted "Try it" widget talk to your agent, opt in explicitly:

WEB_ORIGINS=https://framesleuth.com,https://www.framesleuth.com

The agent stays bound to loopback; CORS only controls which browser origins may read its responses.

Set API_TOKEN for anything beyond a single-user laptop. With a token set, every /v1 endpoint except /v1/healthz requires Authorization: Bearer <token>. CORS won't stop another local process, or a DNS-rebinding page, from sending requests to loopback. A token will. The Docker stack reads it from .env, and publishes the API on 127.0.0.1 only.

Status: backend, pipeline and MCP server are complete.

Questions? Open an issue, or check runbook.md for common ones.

Available Tools

14 tools
analyze_videoA

Analyze any video and return the new report id.

Works on any kind of video — a bug recording, a feature demo, a design walkthrough, a Loom, a phone capture — and distills it into a structured Context Bundle a coding agent can act on (fix a bug, add or change a feature, or build something new).

Args: path: Path to the video file (.mp4/.webm/.mkv/.mov/.avi). repo_root: Repo to ground references against (pass the open workspace). intent: The user's request to act on, e.g. "fix the save button that hangs", "add a dark-mode toggle like the demo shows", or "build this onboarding screen from the walkthrough". It is recorded on the report and shapes the generated action prompt so the calling agent does what the user actually asked. skill: Built-in summary style — one of the names from list_skills (e.g. "summary", "bug_report", "tutorial", "action_items"). Defaults to "summary". system_prompt: A fully custom system prompt for the summary; overrides skill when provided. action: Built-in action mode shaping the fix-prompt — one of the names from list_actions (e.g. "fix", "explain", "triage", "test", "report", "reproduce"). Auto-picked from the classification when omitted. action_prompt: A fully custom action task; overrides action. ctx: The MCP request context, injected by the server; used to stream progress to the client during the (multi-minute) analysis.

Returns the report id, the summary/fix-prompt resource URIs, the resolved action, and the derived suggested_actions menu.

Inputs: path (required), skill, action, intent, +3 more. Returns: object

get_error_evidenceB
Read-onlyIdempotent

Return the timestamped error evidence for a report.

Inputs: report_id (required). Returns: result

get_keyframe_imageA
Read-onlyIdempotent

Return a keyframe image for a report by its index.

Inputs: index (required), report_id (required).

get_reportA
Read-onlyIdempotent

Return the Context Bundle for a report id.

view="full" (default) returns everything; view="slim" returns the action-relevant subset (classification, quality, steps, evidence, candidates, suggested actions) for agents on a small context window.

Typed as a Literal so the client rejects a typo. As a bare str, view="slimm" silently returned the FULL bundle — defeating the only purpose of the parameter for exactly the caller who needed it.

Inputs: report_id (required), view. Returns: object

get_repro_stepsA
Read-onlyIdempotent

Return the numbered reproduction steps for a report.

Inputs: report_id (required). Returns: result

get_suggested_actionsA
Read-onlyIdempotent

Return the machine-readable next-step menu for a report.

Each item is {action, label, rationale, ref} — present them to the user or auto-invoke the referenced resource/tool. Recomputed from the current bundle so it reflects the latest grounding/quality.

Inputs: report_id (required). Returns: result

get_timelineA
Read-onlyIdempotent

Return the merged event timeline for a report.

Inputs: report_id (required). Returns: object

get_video_gifA
Idempotent

Render an animated GIF preview of the video for a report.

Useful for embedding a short looping preview in an issue, chat, or PR description. fps/width/start/end are optional and clamped to safe ranges; the GIF is cached on disk per parameter set.

Inputs: report_id (required), end, fps, start, +1 more.

list_actionsA
Read-onlyIdempotent

List built-in action modes (names + descriptions) for analyze_video.

Inputs: no named parameters. Returns: object

list_reportsA
Read-onlyIdempotent

List all available report ids (from any analyzed video).

Inputs: no named parameters. Returns: result

list_skillsA
Read-onlyIdempotent

List built-in summary skills (names + descriptions) for analyze_video.

Inputs: no named parameters. Returns: object

locate_in_codeA
Read-onlyIdempotent

Return code candidates already grounded in the bundle, or re-ground now.

repo_root is REQUIRED when the bundle carries no candidates yet — it is never inferred from the working directory.

Inputs: report_id (required), repo_root. Returns: result

renderA
Read-onlyIdempotent

Render a report as a shareable artifact.

format is one of markdown, issue (GitHub issue text), or test-plan. Returns the rendered text.

Inputs: report_id (required), format. Returns: result

render_html_videoA
Idempotent

Render an HTML document (CSS / JS / canvas animation) to mp4/gif/webm.

Use this to export a self-contained animated HTML page (e.g. one you just designed) as a shareable clip. Captures the animation frame-by-frame (full color, no dropped frames, no quality loss) and encodes a color-correct H.264 MP4 / VP9 WebM / palette GIF — up to 4K, 5-60 fps. Returns the absolute path to the encoded file, written under the bundle directory. Requires the optional render extra (Playwright) + ffmpeg.

Inputs: html (required), fps, width, format, +2 more. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.1/5.0

    Scored across 14 tools

    Disambiguation5/5

    Each tool targets a distinct resource or action: list_* for enumerating options, analyze_video for creation, get_* for specific report components, render for output formatting, and render_html_video for HTML-to-video conversion. Even the two visual retrieval tools (get_keyframe_image and get_video_gif) are clearly distinguished by static vs. animated output.

    Naming Consistency4/5

    Most tools follow a clear verb_noun pattern (list_*, get_*, analyze_video, render_html_video). The main deviation is the bare verb 'render' which lacks an object, and 'locate_in_code' uses a prepositional structure, but overall the naming is predictable and readable.

    Tool Count5/5

    With 14 tools, the server is well-scoped for a video analysis platform. Each tool serves a distinct purpose in the workflow—analysis, report retrieval, configuration enumeration, and rendering—without unnecessary bloat or redundancy.

    Completeness5/5

    The tool surface covers the full lifecycle from video analysis (analyze_video) through report retrieval (get_report, get_repro_steps, get_error_evidence, etc.) to output generation (render, render_html_video). It also provides supporting tools like list_skills, list_actions, and list_reports to avoid dead ends. No critical gaps are apparent.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers