Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@crv-guide import this pasta video and guide me through it hands-free"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Alexa+ follows a cooking video with you, hands-free, because it actually watched the video.
Cooking from a video is awkward when your hands are covered in flour: the video keeps going, you can't scrub back, a quantity is said once and gone, and you never know if yours looks right. Recipe apps turn the captions into a text recipe. Second Look gives Alexa+ the video itself:
"How much water?" It answers from what was said and when: "about three quarters of a cup to start, then a little at a time; that's at 1:24 in the video", with the evidence on screen.
"What should the dough look like now?" It shows the exact frame from the video for that step and describes details the narrator never says (hand position, thickness, texture).
"Next." "Go back." "Repeat that." One step at a time, read aloud, never skipping ahead on its own.
"Set a timer for the rest." The 30-minute rest from the video becomes a real timer.
"I'm back, where was I?" Close the app, restart everything, come back tomorrow: your step and your timers are still there.
Track: Alexa+ (self-hosted MCP server, spec 2025-11-25, Streamable HTTP) · Mini challenges: AWS Builder (Strands Agents SDK), Open Source.
Demo video: (link added at submission)
┌──────────────────────────── Second Look (simulated Alexa+ Echo Show) ─────────────┐
you ── voice ──▶│ browser: Web Speech API ──▶ /api/turn ──▶ Strands Agent (AWS Strands Agents SDK) │
│ the screen itself is also an MCP client │ MCPClient (Streamable HTTP) │
└───────────────────┬────────────────────────────────┼─────────────────────────────┘
│ MCP: initialize / tools/list / tools/call (protocol 2025-11-25)
▼ ▼
┌──────────────────────── crv-guide MCP server (this repo, server/) ──────────────────┐
│ import_video ─▶ crv watches the video (scene-change keyframes + timestamped transcript)│
│ ─▶ a multimodal model writes the guide from frames AND words, │
│ every step pinned to real frame numbers, times and transcript lines │
│ start_session · get_session · move_step · ask_source · show_moment · set_timer … │
│ SQLite: guides, sessions, step progress, timer deadlines (survives restarts) │
└────────────────────────────────────────────────────────────────────────────────────────┘Two different MCP clients talk to the same server and share its state: the Strands agent (conversation) and the screen (renders cards, handles the Back / Repeat / Next buttons). The MCP trace button on the screen shows every JSON-RPC message live: initialize requesting protocol 2025-11-25, the negotiated version, tools/call with its arguments.
Alexa+ developer tools (MCP Toolkit, Web Simulator) are select-partners only, so, as the hackathon FAQ describes, the device is simulated by a web page that acts as a real MCP client. The server itself is a standard MCP server any MCP client can use.
Tool | What it does |
| Watch a video (URL, local file, or |
| Processing stage, or the finished guide outline (steps, ingredients, durations). |
| Guides already built. |
| Start cooking at step 1. Returns the step card + the step's video frame (image). |
| "Where was I?" Resumes the latest unfinished session, with live timer time-left. |
| Moves only when asked; |
| BM25 search over what was said (timestamped transcript windows) and what each step shows; returns passages + matching frames as images. The assistant answers only from these. |
| The exact frame (grabbed from the source video at that second) + a replayable clip range. |
| Timers stored as absolute deadlines; idempotent with |
Every step in a guide keeps its evidence: the frame number it was written from, its time range, the transcript lines behind each quantity, and a visual_note for something visible in the frames that the narration never says. The guide is validated against the real video before it is stored (frame numbers must exist and fall inside the step, times are clamped, line numbers must exist), so it can only cite things that are in the video.
Related MCP server: DemoMotion
Requirements: Python 3.11–3.13, ffmpeg on PATH, and one model key (OpenAI or Gemini). Tested on macOS (Apple Silicon).
git clone https://github.com/HUANGCHIHHUNGLeo/second-look && cd second-look
python3 -m venv .venv && .venv/bin/pip install -e . # or: uv venv && uv pip install -e .
server/scripts/fetch_sample.sh # sample video for clip replay (CC BY, ~150 MB)
cp .env.example .env # add OPENAI_API_KEY (or GEMINI_API_KEY + SECOND_LOOK_PROVIDER=gemini)
./run.sh # MCP server :8765 + screen :8770Open http://127.0.0.1:8770 in Chrome, click Load the sample video, then tap the mic (or press Space) and talk, or type. The sample's guide ships pre-built in server/fixtures/pasta/, so it loads instantly; paste any other video URL to watch a new one (needs GEMINI_API_KEY for the guide step; a 2–3 minute video takes about a minute to watch, plus the model's time).
Use the server from any MCP client: .venv/bin/crv-guide-mcp --port 8765, then connect to http://127.0.0.1:8765/mcp (Streamable HTTP). Smoke test with a real MCP client: .venv/bin/python server/tests/smoke_client.py.
Before (existing open-source project by the same author): claude-real-video (crv) could extract keyframes and a transcript from a video, with a local stdio MCP server for chat clients.
Built for this hackathon (all of this repo): the crv-guide Streamable HTTP MCP server (spec 2025-11-25) with persistent cooking sessions, step navigation, timers and evidence search; the frames-plus-transcript guide compiler with validation against the source video; exact-moment frame extraction and clip replay; the simulated Alexa+ screen with voice, cards and a live MCP trace; the Strands agent. crv is used unchanged as a library (pinned 0.10.7).
The conversation runs on the Strands Agents SDK (AWS's open-source agent framework), in second_look/agent.py:
strands.tools.mcp.MCPClient(url=...) connects to the crv-guide server over Streamable HTTP and loads its tools at startup (list_tools_sync).
strands.Agent(model=..., tools=..., hooks=[UiEvents()], conversation_manager=SlidingWindowConversationManager(16)) runs the tool loop. An AfterToolCallEvent hook forwards every tool result (step cards, evidence, frames, timers) to the screen.
Model providers through Strands: strands.models.openai.OpenAIModel (default gpt-4.1-mini) or strands.models.gemini.GeminiModel. Inference runs on the provider's API; the agent loop runs locally. No AWS-hosted services are used.
No keys in the repo; everything comes from .env (git-ignored). The screen never sees a key: the agent runs server-side.
The MCP endpoint has DNS-rebinding protection (allowed Host/Origin lists) and a CORS allow-list limited to the screen's origin.
Data stays local (SQLite + files under ~/.crv-guide) except what is sent to the chosen model provider (frames and transcript for the guide; conversation and tool results for the agent).
Not tested on a real Alexa+ device or console: those tools are select-partners only. The simulated screen speaks MCP exactly as described above.
Timers ring while the screen is open; after a restart they show "done" or the correct time left (deadlines are stored), but there are no background notifications.
Voice uses the browser's Web Speech API (Chrome). Typing works everywhere.
Answers are only as good as what the video shows and says; when the search finds nothing the assistant says the video doesn't cover it.
Sample video: "Cooking with Laura: Homemade Pasta" by Northampton Community Television, CC BY 3.0, via Wikimedia Commons. The bundled keyframes and transcript in server/fixtures/pasta/ are derived from it under the same license.
Built with crv / claude-real-video, the MCP Python SDK, Strands Agents, Gemini and OpenAI models.
License: MIT (code). Sample media: CC BY 3.0 as above.
Fetch more keyframes from a video already analysed with watch_video.
Args: source: The same URL or path given to watch_video. start_index: 1-based index of the first frame to return. count: How many frames to return (1-24).
| Name | Required | Description | Default |
|---|---|---|---|
| count | No | ||
| source | Yes | ||
| start_index | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full burden. It usefully discloses that source must match the value passed to watch_video, that indexing is 1-based, and that count is capped at 24, but it says nothing about what happens if the video was never analysed or about the shape of the returned frames.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The leading sentence front-loads the core purpose, and the Args block is compact. The raw docstring formatting ('Args:' with indentation) leaks through and is slightly less scannable than prose, but no sentence is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a three-parameter read tool with no annotations and no output schema, the description should establish what comes back (image data, URLs, thumbnails) and what happens at the end of a video's frame list. The pagination parameters are covered, but the return shape is entirely undocumented anywhere.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, yet the description documents all three parameters with meaning the schema does not carry: the source must be the identical URL/path from watch_video, start_index is 1-based, and count has a 1-24 range. It loses a point only for not restating the default of 12 or the frame-count limits of the underlying video.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Fetch more keyframes from a video') and anchors it to a prerequisite operation (watch_video), which cleanly separates it from the sibling that does the initial analysis. It stops short of the 5 because it never says what a returned frame actually is.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'from a video already analysed with watch_video' gives an explicit precondition and points to the sibling that must run first. There is no statement of when not to use it or of get_transcript as an alternative for non-visual needs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the full timestamped transcript of a video already analysed, without re-sending any frames. Use when you only need the words (quoting, searching a long talk) and the images would waste context.
Args: source: The same URL or path given to watch_video. max_chars: Truncate beyond this many characters (default 20000).
| Name | Required | Description | Default |
|---|---|---|---|
| source | Yes | ||
| max_chars | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full burden and does disclose the key trait: content is served from an existing analysis with no frame re-sending, and max_chars truncates. What is left unstated is the failure mode when the source was never analysed (error vs. implicit fetch) and the exact timestamp format returned.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two front-loaded sentences covering purpose, use case and context economics, followed by a tight two-parameter args block. No sentence is filler and the differentiating claim (no frames re-sent) comes first.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter read tool with no output schema, the description covers purpose, prerequisite, cost rationale, and truncation. It only leaves the not-yet-analysed case and the precise return shape undefined, which are minor for this tool class.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate and largely does: 'source' is constrained to the same URL or path previously passed to watch_video, and max_chars is explained as a truncation threshold with the default. It does not clarify whether source accepts either URL or local path forms interchangeably.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (return) and resource (full timestamped transcript) plus a scope qualifier: a video already analysed. It also implicitly distinguishes itself from get_frames ('without re-sending any frames') and from watch_video ('the same URL or path given to watch_video'), so an agent can route without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit when-to-use condition — quoting or searching a long talk where images would waste context — and states the precondition that the video must already have been analysed. The alternative (get_frames / watch_video) is implied by the context-cost framing rather than named explicitly, keeping this just short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List every video crv has already watched, newest first, with how long it was and how many searchable lines it produced. Cheap way to check whether a video needs watching again before spending time on it.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden and does disclose useful traits: read-only listing, newest-first ordering, the fields returned (duration, count of searchable lines), and a cost profile ("cheap"). It does not mention pagination, result limits, or whether transcripts are included.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core action and scope, with no filler. "How many searchable lines it produced" is slightly opaque phrasing but still earns its place by telling the agent what it gets back.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, no-output-schema list tool the description covers purpose, ordering, payload contents and cost, which is most of what an agent needs to decide to call it. Limits on result size and whether transcripts come back are the remaining gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing for the description to disambiguate; baseline 4 applies. The description instead spends its words on scope and return contents, which is the right allocation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (List) and resource (every video already watched) plus scope, ordering (newest first) and returned fields. The clause about checking whether a video needs watching again implicitly distinguishes it from watch_video without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Cheap way to check whether a video needs watching again before spending time on it" gives a concrete triggering condition and a cost-based rationale versus the heavier watch_video path. It stops short of explicitly naming the alternative tool or stating when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Search everything crv has ever watched — spoken words and on-screen text — and get back the video, the timestamp and the matching line. Use this instead of watch_video when the answer might already be on disk, or when the question spans several videos ("which video mentioned pricing?").
Args: query: What to look for, in any language. limit: Max hits to return (1-50). kind: "any", "speech" (spoken), "onscreen" (text on screen), or "note".
| Name | Required | Description | Default |
|---|---|---|---|
| kind | No | any | |
| limit | No | ||
| query | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does disclose the searchable corpus (everything ever watched), the modalities searched, and what each hit contains. It omits cost/latency expectations and result ordering, which keeps it short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose and the when-to-use rule, then a compact args block. Every line carries information needed given the bare schema, though the args section could be marginally tighter.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description rightly explains the return fields, and it covers all three parameters. Gaps remain around the meaning of the 'note' kind and defaults/pagination, but nothing prevents an agent from calling it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it largely does: it explains query is language-agnostic, gives the limit range (1-50), and enumerates kind values with 'speech' and 'onscreen' glossed. It leaves 'note' unexplained and omits the default for limit and kind.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Search everything crv has ever watched') scoped to both spoken words and on-screen text, and explicitly states the return shape (video, timestamp, matching line). This clearly separates it from siblings like get_transcript and watch_video.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It names the alternative tool ('use this instead of watch_video') and gives two concrete triggering conditions: when the answer may already be on disk, or when the question spans several videos, including an example query. No inference is left to the agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Actually watch a video: download (URL) or read (local path), extract scene-aware deduplicated keyframes and a timestamped transcript, all locally. Returns the transcript plus the first batch of keyframes as images; use get_frames for more frames from the same video.
Args: source: Video URL (YouTube, Instagram, ...) or a local file path. max_frames: How many keyframes to return inline (1-24, default 12). language: Transcript language hint, e.g. "auto", "en", "zh". transcribe: Set false to skip audio transcription (frames only).
| Name | Required | Description | Default |
|---|---|---|---|
| source | Yes | ||
| language | No | auto | |
| max_frames | No | ||
| transcribe | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden and does well: it discloses that processing is local, that keyframes are deduplicated, that only a first batch of images is returned inline, and that transcribe=false degrades to frames-only. It omits failure modes (unsupported URLs, private videos), duration limits, and whether remote downloads require network access, which 'all locally' leaves slightly ambiguous.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The lead sentence front-loads the core behavior and return shape before any parameter detail. The Args block is somewhat redundant in structure but justified given zero schema description coverage; nothing is padded or off-topic.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description correctly explains what comes back (transcript plus inline keyframe images) and how to continue fetching frames. For a four-parameter tool with a network/local duality, missing details on video-length limits, supported URL failures, and total keyframe counts keep it short of complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it does for all four parameters: source (URL for YouTube/Instagram or a local file path), max_frames with range and default (1-24, default 12), language with concrete example values, and transcribe with the semantic consequence of setting it false. This adds real meaning the bare schema titles do not.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb and resource ('Actually watch a video') and enumerates the concrete outputs: scene-aware deduplicated keyframes plus a timestamped transcript. It also distinguishes itself from the sibling get_frames by naming it as the follow-up path for additional frames.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly routes the agent: this tool returns only the first batch, and 'use get_frames for more frames from the same video' names the alternative and the condition that selects it. It gives no guidance on when to prefer get_transcript or list_watched, so the routing is clear but not exhaustive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0get_framesget_transcriptlist_watchedsearch_memorywatch_videoScored across 5 tools
Each tool targets a distinct stage of the pipeline: watch_video ingests, get_transcript and get_frames retrieve derived artifacts, search_memory queries across the whole cache, and list_watched inventories what exists. The descriptions explicitly cross-reference each other and state when NOT to use a tool (e.g. search_memory over watch_video for already-seen content). The transcript-vs-frames split is clearly signposted by modality and context cost.
All five tools use a consistent snake_case verb_noun pattern (get_transcript, watch_video, get_frames, search_memory, list_watched). Retrieval verbs are used consistently: get_ for fetching an artifact from a known source, list_/search_ for exploration, watch_ for the domain's primary action.
Five tools is tightly scoped for a video-ingestion-and-recall server, with each tool earning its place as a distinct step (ingest, frames, transcript, search, inventory). No obvious redundancy or bloat.
The core lifecycle (watch, page frames, get text, search across videos, list cache) is fully covered with no dead ends for the main workflow. Minor gaps: search_memory accepts kind="note" but no tool can write notes, and there is no way to delete/forget or re-index a stale video other than re-watching it.
Voice-led, FSRS-scheduled flashcards from YouTube, PDFs, web, or text. Auto-graded quizzes.
Make how-to videos, social clips and help articles from Claude or ChatGPT, and run your help center.
Transcripts, summaries, chapters and timestamped answers for video links, for AI agents.
Narrated demo and tutorial videos of your web app, from a guide or your coding agent.
