visual
Extract visual context from videos by querying transcripts, auto-detecting demonstrations, or specifying timestamps. Assist mode identifies gaps where user screenshots are needed.
Instructions
ONLY call this tool when the user explicitly asks for visual context, screenshots, or frames. Do NOT call it alongside take_notes or download_audio unless the user specifically requests visuals. Extract visual context from a video at moments that matter. Four modes: (1) Query mode (default): describe what you need visual context for and the tool finds those moments in the transcript, then extracts frames. (2) Auto mode: autonomously detects moments where the speaker implies visual content (UI actions, screen recordings, demonstrations). (3) Manual mode: extract frames at specific timestamps. (4) Assist mode: analyzes the transcript for visual gaps and returns time ranges where the user should provide their own screenshots (ideal for talking-head videos where the speaker describes a UI but doesn't show it). Frames are stored in augent memory alongside the transcription and embedded in the .md file as Obsidian wikilinks.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | Video URL (YouTube, etc). Downloads the video automatically if video_path is not provided. | |
| auto | No | Auto mode: autonomously detect visual moments from transcript (UI actions, demonstrations, spatial references). Default: false. Ignored if query or timestamps are provided. | |
| clear | No | Remove all previously extracted frames and the visual context .md for this video. Use to redo or clean up. Default: false | |
| query | No | What you need visual context for. The tool searches the transcript semantically and extracts frames at matching moments. Examples: 'connecting Gmail to the agent', 'the dashboard configuration', 'where he sets up the branching logic'. | |
| top_k | No | Number of transcript matches to extract frames for in query mode. Default: 10 | |
| assist | No | Assist mode: analyze the transcript for visual gaps and return time ranges where the user should provide their own screenshots. Ideal for talking-head videos or podcasts where the speaker describes a workflow or UI but the video doesn't show it. No frames are extracted — instead, returns structured gaps with time windows, transcript excerpts, and what kind of screenshot would help. Default: false. | |
| max_frames | No | Maximum frames to extract. Default: 30 | |
| model_size | No | Whisper model size for transcription. Default: 'tiny' | |
| timestamps | No | Manual mode: list of timestamps (in seconds) to extract frames at. Overrides query and auto detection. | |
| video_path | No | Path to a video file (MP4, MKV, etc). Can be output from clip_export. | |
| context_words | No | Words of context around each match in query mode. Default: 40 |