This MCP server lets you watch and index videos locally, then retrieve frames, transcripts, and searchable memories.
Watch a video from a URL or local path, extracting scene-aware keyframes and a timestamped transcript.
Fetch additional keyframes from an already-analyzed video.
Search everything previously watched across spoken words, on-screen text, and notes, with video/timestamp/line results.
List all videos already watched, newest first, with duration and searchable-line counts.
Retrieve a full timestamped transcript without re-sending frames.
Provides an Alexa+ add-on experience for hands-free cooking guidance: answers voice questions about a cooking video with timestamped evidence, shows recipe cards and step-by-step instructions on an Echo Show-style screen, replays relevant video moments, and manages cooking timers and session progress.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@crv-guide import this pasta video and guide me through it hands-free"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Alexa+ follows a cooking video with you, hands-free, because it actually watched the video.
Cooking from a video is awkward when your hands are covered in flour: the video keeps going, you can't scrub back, a quantity is said once and gone, and you never know if yours looks right. Recipe tools can turn the captions into a text recipe. Second Look gives Alexa+ the video itself:
"Alexa, how much water did she use?" It answers from what was said and when: "about three quarters of a cup of cold water to start; she says it at 1:24", with the evidence on screen and the moment ready to replay.
"Alexa, what do I need for this?" The recipe on one card, compiled from the video: ingredients with amounts and the steps. Each amount she actually says carries the moment she says it (tap to hear it); amounts the video never says are marked as estimated or not said, instead of being dressed up as hers.
"Alexa, how much sugar?" "The video never mentions sugar." An amount question about something the video never mentions gets no evidence back, so Alexa says so instead of reading out the nearest other number.
"Show me how she makes the shape with her thumb." It finds the moment she describes the move, shows the frame a beat later and replays the clip, and describes details the narrator never says (hand position, thickness, texture).
"Next." "Go back." "Repeat that." One step at a time, read aloud, never skipping ahead on its own.
"Set a timer for the rest." The 30-minute rest from the video becomes a real timer.
"I'm back, where was I?" Close the app, restart everything, come back later: your step and your timers are still there.
Hands-free mode. Like an Echo Show, the screen keeps listening and acts only on sentences that start with "Alexa". It closes the microphone while Alexa talks, while the agent works and while a clip plays (the view reports when playback starts and stops), so it doesn't take its own voice or the cook in the video for a request.
Track: Alexa+ (self-hosted MCP server, spec 2025-11-25, Streamable HTTP) · Mini challenges: AWS Builder (Strands Agents SDK), Open Source.
Demo video: https://youtu.be/5i-394l3xQk
you ── "Alexa, ..." ──┐
▼
┌──────────────── simulated Alexa+ host (web/, an Echo Show 15 class screen) ────────────────┐
│ voice (Web Speech API, hands-free wake word) ─▶ /api/turn ─▶ Strands agent ─┐ │
│ add-on view: ui://second-look/cook.html in a sandboxed iframe │ MCPClient │
│ ▲ MCP Apps over postMessage: ui/initialize, ui/notifications/tool-result, │ (Streamable │
│ │ tools/call proxied by the host │ HTTP) │
│ └──── host: MCP client + MCP Apps host ───────────────────────────────────┤ │
└──────────────────────────────┬──────────────────────────────────────────────┼──────────────┘
│ MCP 2025-11-25: initialize · tools/list · resources/read · tools/call
▼ ▼
┌──────────────────────────── crv-guide MCP server (server/) ─────────────────────────────────┐
│ import_video ─▶ crv watches the video (scene-change keyframes + timestamped transcript) │
│ ─▶ a multimodal model writes the guide; checks.py checks its numbers │
│ start_session · get_session · move_step · ask_source · show_moment · set_timer … │
│ (session tools carry _meta.ui.resourceUri → the ui:// view, text/html;profile=mcp-app) │
│ SQLite: guides, sessions, step progress, timer deadlines (survives restarts) │
└────────────────────────────────────────────────────────────────────────────────────────────┘The add-on UI is built the way Alexa+ renders add-ons. Amazon's Alexa+ add-on guide describes an add-on's screen as a self-contained HTML bundle that Alexa shows in a sandboxed iframe and talks to over the MCP Apps postMessage bridge (Layout and Rendering). So the server registers one ui://second-look/cook.html resource (MIME text/html;profile=mcp-app, with its CSP in _meta.ui) through the MCP Python SDK's Apps extension, and the session tools point at it with _meta.ui.resourceUri. The view is authored on Amazon's 768 px base canvas (Block viewport, scaled 1.667x on an Echo Show 8/15) with the guide's type, spacing, radius and dark card colour tokens (Visual Foundations). Text-only MCP clients simply ignore it.
The simulated device is an MCP Apps host. It reads the resource, mounts it in a sandbox="allow-scripts" iframe, answers ui/initialize with a host context (theme, container size, device class), forwards every tool result from the agent's turn as ui/notifications/tool-result, and proxies the view's own tools/call (its Back / Next buttons, evidence cards) to the server. When Alexa answers with a moment, the host shows it after she has finished speaking; the view shows the frame, plays the clip and returns to the step by itself.
Plain navigation ("next", "go back", "repeat", "I'm back, where was I?") is routed straight to move_step / get_session, the way a voice platform maps simple intents, so it answers instantly and can't be misread by a model. Everything open-ended ("how much water?", "skip to the shaping", "show me the thumb move") goes to the Strands agent, which always gets the screen's current session and step outline with the request. Session tools also return a short voice-ready line (speak, like a skill's output speech), so Alexa says one or two sentences while the view shows the full card.
Two different MCP clients talk to the same server and share its state: the Strands agent (conversation) and the host (screen and add-on view). The MCP trace button shows both kinds of traffic live, in time order: MCP over Streamable HTTP (initialize with protocol 2025-11-25, resources/read of the view, tools/call with arguments) and MCP Apps over postMessage (ui/initialize, ui/notifications/tool-result, the view's tools/call).
Alexa+ developer tools (MCP Toolkit, Web Simulator) are select-partners only, so, as the hackathon FAQ allows, the device is simulated. The server is a standard MCP server any MCP client can use, and its view follows the MCP Apps spec (tested in our simulated host only).
Tool | What it does |
| Watch a video (URL, local file, or |
| Processing stage, or the finished guide outline (steps, ingredients, durations). |
| Guides already built. |
| Start cooking at step 1. Returns the step card + the step's video frame (image). |
| "Where was I?" Resumes the latest unfinished session, with live timer time-left. |
| Moves only when asked; |
| BM25 search over what was said (timestamped transcript windows) and what each step shows; each passage is pinned to the one transcript line that answers the question ( |
| The exact frame (grabbed from the source video at that second) + a replayable clip range. |
| The recipe on one card: ingredients with amounts, then the steps. An amount the narration states carries the line and time it is said (tap to replay); an amount the video never says is marked as estimated, a missing one as not said. |
| Timers stored as absolute deadlines; idempotent with |
Every step in a guide keeps its evidence: the frame number it was written from, its time range, the transcript lines behind each quantity, and a visual_note for something visible in the frames that the narration never says. The guide is checked against the real video before it is stored: frame numbers must exist and fall inside the step, times are clamped, line numbers must exist, and every number in an instruction (amounts, durations, temperatures) must appear in the narration the step cites or the lines spoken during it, with the same unit and, for an amount of something, next to the same ingredient word: "two cups of sugar" does not pass on "two teaspoons of salt" or "two cups of flour" (checks.py). Numbers that aren't said are flagged on the card ("Not said in the video"), and a timer is only offered for a duration that is said. It is a lexical check of numbers, units and nearby words, not a proof that an instruction is right. On the sample, one number in the steps isn't in the narration: "four pieces", which she describes as "cut it in half and then half again".
Related MCP server: DemoMotion
Requirements: Python 3.11–3.13, ffmpeg on PATH, and one model key (OpenAI or Gemini). Tested on macOS (Apple Silicon).
git clone https://github.com/HUANGCHIHHUNGLeo/second-look && cd second-look
python3 -m venv .venv && .venv/bin/pip install -e . # or: uv venv && uv pip install -e .
server/scripts/fetch_sample.sh # sample video for clip replay (CC BY, ~150 MB)
cp .env.example .env # add OPENAI_API_KEY (or GEMINI_API_KEY + SECOND_LOOK_PROVIDER=gemini)
./run.sh # MCP server :8765 + screen :8770Open http://127.0.0.1:8770 in Chrome, click Load the sample video, then tap the mic (or press Space) and talk, or type. The sample's guide ships pre-built in server/fixtures/pasta/, so it loads instantly; paste any other video URL to watch a new one (after a video is loaded, New video in the top bar brings the link box back) (the guide step uses Gemini when GEMINI_API_KEY is set, otherwise OpenAI vision; in our tests on deadline night a 2.5-minute video took 2 to 12 minutes end to end).
Tests (install the test extra first: .venv/bin/pip install -e '.[test]' && .venv/bin/playwright install chromium): .venv/bin/python -m pytest server/tests/test_core.py (guide validation and number checks against the video, evidence search, ingredients the video never mentions, sessions, revision checks, idempotent timers) .venv/bin/python web/tests/handsfree_check.py (hands-free mode in a real browser with a scripted recognizer) and .venv/bin/python web/tests/newvideo_check.py (switching to another video after the sample).
Use the server from any MCP client: .venv/bin/crv-guide-mcp --port 8765, then connect to http://127.0.0.1:8765/mcp (Streamable HTTP). Smoke test with a real MCP client: .venv/bin/python server/tests/smoke_client.py.
Before (existing open-source project by the same author): claude-real-video (crv) could extract keyframes and a transcript from a video, with a local stdio MCP server for chat clients.
Built for this hackathon (all of this repo): the crv-guide Streamable HTTP MCP server (spec 2025-11-25) with persistent cooking sessions, step navigation, timers and evidence search; the frames-plus-transcript guide compiler with validation against the source video; exact-moment frame extraction and clip replay; the simulated Alexa+ screen with voice, cards and a live MCP trace; the Strands agent. crv is used unchanged as a library (pinned 0.10.7).
The conversation runs on the Strands Agents SDK (AWS's open-source agent framework), in second_look/agent.py:
strands.tools.mcp.MCPClient(url=...) connects to the crv-guide server over Streamable HTTP and loads its tools at startup (list_tools_sync).
strands.Agent(model=..., tools=..., hooks=[UiEvents()], conversation_manager=SlidingWindowConversationManager(16)) runs the tool loop. An AfterToolCallEvent hook forwards every tool result (step cards, evidence, frames, timers) to the screen.
Model providers through Strands: strands.models.openai.OpenAIModel (default gpt-4.1-mini) or strands.models.gemini.GeminiModel. Inference runs on the provider's API; the agent loop runs locally. No AWS-hosted services are used.
No keys in the repo; everything comes from .env (git-ignored). The screen never sees a key: the agent runs server-side.
The MCP endpoint has DNS-rebinding protection (allowed Host/Origin lists) and a CORS allow-list limited to the screen's origin.
Data stays local (SQLite + files under ~/.crv-guide) except what is sent to the chosen model provider (frames and transcript for the guide; conversation and tool results for the agent).
Not tested on a real Alexa+ device or console: those tools are select-partners only. The simulated host speaks MCP and MCP Apps as described above; the add-on view has only been tested in that host. If a browser blocks autoplay, the clip shows its play button instead.
Timers ring while the screen is open; after a restart they show "done" or the correct time left (deadlines are stored), but there are no background notifications.
Voice uses the browser's Web Speech API (Chrome). Typing works everywhere. Hands-free mode was tested in Chromium with a scripted speech recognizer (web/tests/handsfree_check.py), not yet in a noisy kitchen; real recognition will sometimes mishear.
Answers are only as good as what the video shows and says; when the search finds nothing the assistant says the video doesn't cover it. Answer wording comes from a model: the prompt tells it to use only the numbers in the quoted transcript line, but the spoken answer itself is not re-checked.
Tested on two narrated cooking videos so far (the sample and a goat-chops recipe). The guide writer is written for cooking. Importing a 2.5-minute video took between 2 and 12 minutes on deadline night. Each video is watched once: importing a link whose import has finished returns the stored guide at once (store.guide_by_source_key), so on a shared server a popular video is instant after its first import (two first imports at the same moment are not merged). The first import on a machine also downloads crv's speech-recognition model.
Videos with no speech are not supported yet (the guide compiler needs a transcript). With little narration, steps still come from the frames as well as the words and say what is only visible in the picture, but amounts and times can only be answered from what is said.
Clips: the server downloads the video to cut clips, which is fine for the Creative Commons samples here. A public release would play the moment through the platform's own embedded player (start and end seconds) and keep only timestamps, short quotes and the guide.
Sample video: "Cooking with Laura: Homemade Pasta" by Northampton Community Television, CC BY 3.0, via Wikimedia Commons. The bundled keyframes and transcript in server/fixtures/pasta/ are derived from it under the same license.
Video imported live in the demo: "Goat Chops Recipe Grill Pan | How To Cook Goat Chops Recipe" by Tabs Yummy Recipes, CC BY 3.0, via Wikimedia Commons.
Built with crv / claude-real-video, the MCP Python SDK, Strands Agents, Gemini and OpenAI models.
License: MIT (code). Sample media: CC BY 3.0 as above.
Fetch more keyframes from a video already analysed with watch_video.
Args: source: The same URL or path given to watch_video. start_index: 1-based index of the first frame to return. count: How many frames to return (1-24).
Inputs: source (required), count, start_index.
Return the full timestamped transcript of a video already analysed, without re-sending any frames. Use when you only need the words (quoting, searching a long talk) and the images would waste context.
Args: source: The same URL or path given to watch_video. max_chars: Truncate beyond this many characters (default 20000).
Inputs: source (required), max_chars.
List every video crv has already watched, newest first, with how long it was and how many searchable lines it produced. Cheap way to check whether a video needs watching again before spending time on it.
Inputs: no named parameters.
Search everything crv has ever watched — spoken words and on-screen text — and get back the video, the timestamp and the matching line. Use this instead of watch_video when the answer might already be on disk, or when the question spans several videos ("which video mentioned pricing?").
Args: query: What to look for, in any language. limit: Max hits to return (1-50). kind: "any", "speech" (spoken), "onscreen" (text on screen), or "note".
Inputs: query (required), kind, limit.
Actually watch a video: download (URL) or read (local path), extract scene-aware deduplicated keyframes and a timestamped transcript, all locally. Returns the transcript plus the first batch of keyframes as images; use get_frames for more frames from the same video.
Args: source: Video URL (YouTube, Instagram, ...) or a local file path. max_frames: How many keyframes to return inline (1-24, default 12). language: Transcript language hint, e.g. "auto", "en", "zh". transcribe: Set false to skip audio transcription (frames only).
Inputs: source (required), language, max_frames, transcribe.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 5 tools
Each tool targets a distinct stage of the pipeline: watch_video ingests, get_transcript and get_frames retrieve derived artifacts, search_memory queries across the whole cache, and list_watched inventories what exists. The descriptions explicitly cross-reference each other and state when NOT to use a tool (e.g. search_memory over watch_video for already-seen content). The transcript-vs-frames split is clearly signposted by modality and context cost.
All five tools use a consistent snake_case verb_noun pattern (get_transcript, watch_video, get_frames, search_memory, list_watched). Retrieval verbs are used consistently: get_ for fetching an artifact from a known source, list_/search_ for exploration, watch_ for the domain's primary action.
Five tools is tightly scoped for a video-ingestion-and-recall server, with each tool earning its place as a distinct step (ingest, frames, transcript, search, inventory). No obvious redundancy or bloat.
The core lifecycle (watch, page frames, get text, search across videos, list cache) is fully covered with no dead ends for the main workflow. Minor gaps: search_memory accepts kind="note" but no tool can write notes, and there is no way to delete/forget or re-index a stale video other than re-watching it.
Search, save, organize, cook, and share recipes with any AI assistant.
Persistent food journal for AI assistants. Log meals, save recipes, plan meals and track nutrition.
Recipe search, ingredient matching, nutrition filters and a private cookbook for any AI agent.
Save, import and edit recipes as interactive cards in ChatGPT and Claude, synced to your iPhone.
