Understand video
understand_videoAnalyze videos from page URLs, direct media links, or local files to answer questions about visible content, speech, and on-screen text, returning structured timelines, transcripts, and summaries.
Instructions
Watch a video and answer a question about it, using a video-capable model behind the configured OpenAI-compatible endpoint. source accepts a page URL (TikTok, YouTube, X, Instagram, ... resolved with yt-dlp), a direct media URL with optional headers such as Referer from a scraper, or an absolute local path inside an allowed directory. Returns answer, summary, speech (transcript in the original language), on_screen_text, a timeline in source-video seconds, and analysis metadata with warnings. Videos longer than the duration limit need start_seconds/end_seconds; a short range also gives more detail for fast motion such as workouts. fps is a hint applied only when the request format supports it. Errors carry a code (e.g. ENDPOINT_UNREACHABLE, LIMIT_EXCEEDED, SOURCE_NOT_ALLOWED, MEDIA_NOT_PROCESSED) as a [CODE] text prefix and in _meta.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| fps | No | Frame sampling rate hint. Applied only when the endpoint's request format supports it; a short range gives more detail for fast motion | |
| source | Yes | Video to analyze: a page URL (TikTok, YouTube, X, ...), a direct media URL, or an absolute local path inside an allowed directory | |
| headers | No | Request headers for a direct media URL, such as Referer or Cookie returned by a scraper. Sent only to that URL's host | |
| question | No | What to find out about the video. Omit for a general description | |
| end_seconds | No | Analyze up to this second of the source video | |
| start_seconds | No | Analyze from this second of the source video |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| answer | Yes | ||
| speech | Yes | ||
| summary | Yes | ||
| analysis | Yes | ||
| timeline | Yes | ||
| on_screen_text | Yes |