Enables analysis of public Instagram Reels, providing timestamped transcripts, optional visual understanding, unified search across speech and on-screen content, and moment-level lookup.
Enables analysis of public TikTok videos, including speech transcription, visual observations, search across spoken and shown content, and timestamped moment retrieval.
Supports analysis of YouTube videos and Shorts, offering transcription, optional visual understanding, cross-modal search, moment finding, and a video map of entities and references.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@moreel Check this YouTube Short and tell me what on-screen text says about pricing"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Most video-to-transcript tools stop at the audio. Moreel treats the video itself as the source of truth: it transcribes what was said, reads what was shown (on-screen text, slides, charts, products), and resolves what a vague "this one" or a pointing gesture actually meant — then makes all of it searchable and jumpable, down to the exact second.
It's built to be used three ways:
As a web app — paste a URL, get video + transcript side by side, with in-transcript search and a "what did I miss?" panel for anything shown but never said out loud.
As an HTTP API — the same pipeline, for your own backend/integration.
As an MCP server — a set of tools (understand_video, search_video,
find_moment, get_video_map, ...) so an AI agent can query a video
without ever "watching" it or scrubbing through a transcript by hand.
Transcription — accurate, timestamped speech-to-text (OpenAI Whisper), adaptively re-chunked from word-level timestamps so a segment reflects how the person actually spoke, not an arbitrary decode boundary.
Visual understanding (opt-in) — bounded frame sampling + a vision model surfaces on-screen text, slides, charts, products, and scene context, kept as a separate layer from the transcript, never merged into it.
The Video Map (opt-in, layered on visual understanding) — resolves "this"/"that"/"this one" and pointing/showing/holding gestures to a specific visual entity, with a confidence level and never a fabricated target when the evidence is weak.
Unified search — one query surfaces matches across speech, on-screen text, and resolved references, each labeled by source and one click from seeking the player. Lexical by default; opt-in embeddings add semantic matching (a query for "cost" also finds on-screen text that says "pricing").
"What did I miss?" — surfaces exactly what was visible but never said, classified (on-screen text, visual context, an unspoken visual reference, ...), each anchored to a timestamp — never a generic summary.
Supported sources: public Instagram Reels, TikTok videos, and YouTube videos/Shorts. Private, login-gated, or otherwise access-controlled content is never attempted — it fails with a typed error by design.
Related MCP server: videoseek-mcp
Requirements:
Node.js >= 22
yt-dlp on PATH (or set YTDLP_PATH)
ffmpeg on PATH (or set FFMPEG_PATH)
An OpenAI API key (transcription; also vision/Video Map/embeddings if you enable those)
git clone https://github.com/medaharrat/moreel.git
cd moreel
npm install
cp .env.example .env
# edit .env and set OPENAI_API_KEY
cd web && npm install && cd ..Quickest path — Docker Compose (API + web UI + Postgres + Redis):
export OPENAI_API_KEY=sk-...
# optional: export VISION_ENABLED=true VIDEO_MAP_ENABLED=true
docker compose -f docker-compose.prod.yml up --buildWeb UI at http://localhost:3001, API at http://localhost:8080.
From source, without Docker:
npm run build && npm run migrate:up # Postgres schema (optional — see below)
npm run dev:http # HTTP API + web-facing endpoints, from source
cd web && npm run dev # web UI (separate terminal)
npm run dev # or: the MCP server over stdio, for an agent clientPostgres/Redis are optional in development — without DATABASE_URL, video
records fall back to an in-process cache; without REDIS_URL, rate
limiting/dedup fall back to in-memory, single-replica-only behavior.
All configuration is via environment variables — see .env.example for the
full, documented list. The important ones:
Variable | Default | Purpose |
| required | Transcription (and vision/Video Map/embeddings, if enabled) |
|
| On-screen text/scene understanding alongside the transcript |
|
| Resolve "this"/"that"/pointing references (requires |
|
| Semantic matching on top of lexical search |
|
| Reject media larger than this |
|
| Reject media longer than this |
|
| Bounded concurrency; excess fails fast with |
|
| Cache processed videos (in-memory, or Postgres/Redis if configured) |
| (none) | Enables durable, cross-replica video storage + accounts/API keys |
| (none) | Enables distributed rate limiting/dedup |
|
| Structured log verbosity |
Never hardcode secrets — always via environment/.env.
Point any MCP client at node dist/mcp/server.js (stdio transport):
{
"mcpServers": {
"moreel": {
"command": "node",
"args": ["/absolute/path/to/moreel/dist/mcp/server.js"],
"env": { "OPENAI_API_KEY": "sk-..." }
}
}
}Tool | Does |
| Speech only — timestamped transcript, opt-in visual observations |
| Speech + visual understanding, always on — entry point for most agent use |
| Query across speech, on-screen text, and resolved references |
| The single best timestamped answer to a specific question |
| The full chronological merge of every modality |
| Entities, interactions, and references the Video Map resolved |
| Everything tied to one specific entity (every mention, every moment) |
| The underlying evidence behind one specific map fact, by id |
search_video/find_moment/get_video_map operate on a video_id
returned by a prior transcribe_video/understand_video call — an agent
watches a video once, then queries it repeatedly without re-processing.
Every response is structured JSON — timestamps, confidence, and observed/inferred/uncertain evidence levels — never prose an agent has to parse.
INGEST src/providers/ VideoProvider interface; Instagram/TikTok/YouTube
(yt-dlp), each behind a circuit breaker + rate limit
↓
MEDIA src/media/ SSRF-guarded download, audio extraction, bounded
frame sampling (periodic + scene-change)
↓
TRANSCRIPTION src/transcription/ Transcriber interface; OpenAI Whisper; word-timestamp
re-chunking; hallucination/confidence normalization
↓
VISION src/vision/ VisionProvider (on-screen text/scene) +
VideoInteractionAnalyzer (the Video Map), opt-in
↓
APPLICATION src/app/ Timeline merge, lexical+semantic search, "what did I
miss", video-map resolution — orchestration only,
never provider-specific
↓
TRANSPORTS src/mcp/ MCP tools (stdio)
src/http/ Web UI's API + REST-ish JSON routessrc/domain/ holds the shared types (Transcript, VisualObservation,
VideoRecord, the Video Map's Interaction/LinguisticReference) and a
typed MoreelError taxonomy — nothing above it knows which platform or
which model vendor produced the data. Adding a platform means one more
VideoProvider; adding a model vendor means one more Transcriber/
VisionProvider — nothing else changes.
Cross-cutting concerns live alongside, not inside, the pipeline:
src/cache/, src/observability/, src/usage/, src/config/, src/util/.
URL validation & SSRF protection — every input URL is parsed and
rejected if malformed, non-http(s), or pointing at a private/loopback/
link-local IP (including the cloud-metadata address); redirects are
re-validated the same way (src/media/downloader/ssrf.ts).
No shell interpolation — yt-dlp/ffmpeg run via execFile with
argument arrays, never a shell.
No path traversal — downloaded files live under a server-generated,
randomly named per-request temp directory, always cleaned up in a
finally block, regardless of success or failure.
Bounded everything — size/duration caps enforced while streaming,
hard timeouts on every network/subprocess call, bounded concurrency that
fails fast (RATE_LIMITED) instead of queuing unboundedly.
No secret leakage — typed errors carry a client-safe message; stack traces, file paths, and provider internals are logged server-side only.
Bounded-retention storage, not an archive — processed videos (when
persisted to Postgres/Redis) carry an explicit TTL and are treated as a
cache, matching the same minimal-retention principle as the in-process
cache. See docs/privacy.md.
npm run dev # MCP server from source (tsx), no build step
npm run dev:http # HTTP API from source
npm run typecheck # tsc --noEmit
npm run lint # eslint
npm run format # prettier --writenpm test # everything
npm run test:unit # provider selection, media limits, normalization,
# timeline/search/video-map resolution, config, ...
npm run test:integration # full pipeline, real Postgres video-store round-trips
npm run test:protocol # real MCP Client <-> Server over an in-memory transport
npm run test:regression # transcription quality regression corpusAll network access and subprocess execution (yt-dlp, ffmpeg, the OpenAI
API) are mocked at their interface boundaries in unit/protocol tests — the
integration suite's Postgres-backed tests are the exception, and skip
automatically when DATABASE_URL isn't reachable.
Public content only, by design — Moreel never bypasses login, CAPTCHAs, or other access controls.
Automatic transcription/visual analysis is best-effort, not
human-verified — surfaced via low_confidence/evidenceLevel rather
than silently guessed.
No speaker diarization — multi-speaker audio transcribes in order without speaker labels (Whisper doesn't expose this).
The Video Map doesn't track entity identity via visual similarity across a whole video — it relies on the model reusing a consistent label for a recurring entity within one analysis pass.
Video Map events don't yet carry their own evidence frame thumbnail (timestamp + confidence still make them fully traceable via the player).
Issues and PRs are welcome. npm run typecheck && npm run lint && npm test
should pass before opening one — CI runs the same checks (plus
integration/protocol/regression suites against real Postgres+Redis service
containers) on every PR.
MIT — see LICENSE.
Returns the single best timestamped piece of evidence in a previously-understood video for a specific question — precise, verifiable, and anchored to one moment, unlike search_video which returns every match.
Requires a "video_id" from a prior understand_video or transcribe_video call. Use this when you need one authoritative answer with proof (e.g. "does the creator show pricing anywhere?", "what is he pointing at when he says 'this one'?") rather than a list of every mention. When the best match resolves to a specific visual entity (a pointing gesture, a "this"/"that" reference), the "interaction" field gives the structured target — absent when evidence was too weak to confidently resolve one, never a fabricated guess.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | The specific thing to find — e.g. "pricing", "the revenue slide", "the dashboard". | |
| video_id | Yes | A video id previously returned by understand_video or transcribe_video. |
| Name | Required | Description |
|---|---|---|
| text | No | The exact evidence — what was said or what was shown. |
| found | Yes | False when nothing in the video matched the query at all. |
| query | Yes | |
| answer | No | |
| source | No | |
| frame_id | No | When present, fetch the evidence frame via GET /media/:frame_id on the HTTP API. |
| video_id | Yes | |
| timestamp | No | Seconds from the start of the video where the best evidence occurs. |
| confidence | No | |
| interaction | No | |
| end_timestamp | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description adds meaningful behavioral detail about the interaction field: it is present only when a visual entity is strongly resolved, absent when evidence is weak, and never fabricated. This goes beyond the annotations and adds trust-indicating transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a compact set of four sentences, each carrying distinct information: purpose/differentiation, dependency, usage guidance/example, and interaction field reliability. No filler or redundancy. The most essential info is front-loaded in the opening sentence.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has only two parameters, a 100% schema-described input schema, and an output schema to describe return values. The description sufficiently explains prerequisites, intended use cases, and edge-case behavior (weak evidence), making it equally complete. No critical operational element needing explanation is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 100% schema description coverage, the baseline is 3. The description adds extra value by explaining the query parameter's semantic role (specific things to find, with examples like 'pricing') and stating that video_id must come from prior understand_video or transcribe_video output. This contextual relationship is not in the schema and helps agents compose calls correctly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Returns') and a precise resource ('single best timestamped piece of evidence'), and explicitly contrasts itself with the sibling search_video ('unlike search_video which returns every match'). An agent can clearly distinguish this tool from its siblings without inspecting the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states a prerequisite (requires a video_id from prior understand_video or transcribe_video), provides usage examples, and gives a clear selection criterion: 'use this when you need one authoritative answer with proof rather than a list of every mention'. The 'when not to use' is implied by the contrast with search_video, giving an exclusionary signal.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Returns everything the Video Map knows about one specific entity (a person/product/object) from get_video_map — every interaction it was involved in and every linguistic reference resolved to it, in chronological order.
Use this after get_video_map or search_video/find_moment surfaces an entity id, to see its full history across the video (e.g. "every time the product was pointed at or referred to") rather than one isolated moment.
| Name | Required | Description | Default |
|---|---|---|---|
| video_id | Yes | A video id previously returned by understand_video or transcribe_video. | |
| entity_id | Yes | An entity id from get_video_map, or a target_entity_id from search_video/find_moment. |
| Name | Required | Description |
|---|---|---|
| type | No | |
| found | Yes | |
| label | No | |
| video_id | Yes | |
| entity_id | Yes | |
| last_seen | No | |
| confidence | No | |
| first_seen | No | |
| references | Yes | Every linguistic reference resolved to this entity, chronologically. |
| description | No | |
| interactions | Yes | Every interaction this entity was the actor or target of, chronologically. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already establish that the tool is read-only, idempotent, and non-destructive, so there is no contradiction. The description adds meaningful behavioral detail beyond those annotations: results include every interaction and every linguistic reference resolved to the entity, returned in chronological order. This tells the agent what kind of data to expect and how it is organized.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two compact sentences with no repetition or fluff. It front-loads the core function, then provides usage context and a concrete example in parentheses. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter lookup tool with comprehensive annotations and an output schema, the description is complete: it defines what the tool returns, the prerequisites, the input IDs, the output scope, and the ordering. An agent has everything it needs to call this tool correctly and understand the result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and both parameters already have clear descriptions explaining where the IDs come from (understand_video/transcribe_video for video_id; get_video_map or search_video/find_moment for entity_id). The tool description reinforces this relationship but does not add new parameter-level semantics beyond the schema, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb and resource: it 'Returns everything the Video Map knows about one specific entity' and clarifies that entity means 'a person/product/object'. It also differentiates itself from siblings like get_video_map and search_video/find_moment by emphasizing the full chronological history rather than an isolated moment.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use the tool: 'Use this after get_video_map or search_video/find_moment surfaces an entity id.' It further clarifies the use case—full history across the video—and contrasts it with seeing 'one isolated moment,' which helps the agent choose between this and more narrowly scoped tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Looks up the full evidence behind a single Video Map fact by id — never invents detail beyond what was actually recorded (see get_video_map, get_video_entity). Use this to double-check a specific claim before treating it as authoritative, e.g. confirming a reference's evidence_level before repeating its resolved target as fact.
| Name | Required | Description | Default |
|---|---|---|---|
| video_id | Yes | A video id previously returned by understand_video or transcribe_video. | |
| evidence_id | Yes | An entity/interaction/reference id, from get_video_map, get_video_entity, search_video, or find_moment. |
| Name | Required | Description |
|---|---|---|
| kind | No | |
| found | Yes | |
| detail | No | The full underlying record for this id. |
| modality | No | What kind of fact this is — an entity type, interaction type, or reference relation. |
| video_id | Yes | |
| timestamp | No | |
| confidence | No | |
| evidence_id | Yes | |
| evidence_level | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds value beyond annotations by stating that the tool 'never invents detail beyond what was actually recorded', which is a behavioral guarantee. Annotations already provide readOnlyHint=trueتریidempotentHint=true and destructiveHint=false, but the description adds specific context about data fidelityesternity. It does not contradict annotations and accurately reflects a read-only lookup operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, with two sentences. The first sentence clearly states the purpose and constraint, and the second provides a concrete use case. It is front-loaded with the core function and behavioral guarantee. It could be slightly more structured, but it is effective and does not waste words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity, with 2 required parameters and an output schema, the description covers the essential aspects: what it does, when to use it, and behavioral guarantees. It mentions the ids needed and where they come from in the schema. The description is sufficient for an agent to understand when and how to use this tool correctly, especially with the output schema and annotations providing additional context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema fully describes both parameters (video_id and evidence_id) with explanations of where they come from. The description does not add additional parameter semantics beyond what the schema provides. Since schema description coverage is 100%, the baseline score of 3 is appropriate; the description does not need to repeat the parameter details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: to look up evidence for a single Video Map fact by id. It uses a specific verb ('looks up') and the resource ('full evidence behind a single Video Map fact'). It differentiates itself from siblings by explicitly referencing get_video_map and get_video_entity, making it clear that this tool retrieves evidence details rather than the map or entity data itself.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit guidance on when to use this tool: to double-check a specific claim before treating it as authoritative, such as confirming a reference's evidence_level. It also implies when not to use it (e.g., when you need the full map or entity, use the mentioned siblings). The mention of alternative tools (get_video_map, get_video_entity) provides clear routing, and the use case ('double-check a specific claim') is well-defined.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Returns the Video Map: the semantic layer connecting what was SAID to what was VISIBLE and what the person DID — entities (people/products/objects that recur across the video), interactions (points_at, shows, holds, etc.), and references (what "this"/"that one" actually meant).
This is what resolves "speech → reference → visual target" — a transcript alone cannot tell you what "this one" refers to; this can, when the visual evidence was strong enough. Every interaction/reference has an "evidence_level" (observed/inferred/uncertain) and a "confidence" — absent "target_entity_id" means the evidence was too weak to confidently resolve a target, which is the CORRECT and expected answer in that case, not a failure.
Requires a "video_id" from a prior understand_video call with visual analysis enabled. Returns empty arrays (not an error) when the Video Map feature wasn't enabled or found nothing — use search_video/find_moment for the underlying speech/visual search either way.
| Name | Required | Description | Default |
|---|---|---|---|
| video_id | Yes | A video id previously returned by understand_video or transcribe_video. |
| Name | Required | Description |
|---|---|---|
| scenes | Yes | |
| entities | Yes | Every distinct person/product/object the analysis identified, with a stable id reusable across interactions and references. |
| video_id | Yes | |
| references | Yes | Linguistic references ("this", "that one") resolved to a specific entity, when evidence was sufficient. |
| interactions | Yes | Things the person visibly did — pointing, showing, holding, etc. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnlyHint, idempotentHint, and destructiveHint=false. The description adds critical behavioral detail: evidence_level ('observed/inferred/uncertain'), the meaning of absent target_entity_id (expected, not failure), and the empty-array vs error contract. These go far beyond the structured annotations and reduce deployment surprises.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is packed with purpose, usage, semantics, and edge cases, and every sentence earns its place. It front-loads what the tool does, then gives context and fallback guidance, ending with the clearest routing alternative. No fluff or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only look-up tool with a single parameter and a deep output schema, the description covers the only input prerequisite, the meaning of key output fields, the edge case for un-resolved references, and the fallback behavior when data is absent. Nothing an agent needs to safely invoke and interpret this tool is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes video_id ('A video id previously returned by understand_video or transcribe_video'), providing full coverage of the single parameter. The description adds one valuable nuance: 'with visual mapping enabled', which is not in the schema and is essential for determining when the tool will return meaningful results.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Returns') and names a concrete resource ('the Video Map: the semantic layer') and explains what it connects (speech, visible entities, interactions, references). It clearly distinguishes itself from siblings like search_video or find_moment by stating it resolves speech-to-reference mappings, not raw search.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit preconditions ('Requires a video_id from a prior understand_video call with visual analysis enabled'), explicit fallback behavior ('Returns empty arrays when the feature wasn't enabled or found nothing'), and explicit alternatives ('use search_video/find_moment for the underlying speech/visual search either way').
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Returns the full chronological, merged timeline of a previously-understood video — every spoken segment interleaved with every visual observation, in time order. This is the underlying structure search_video and find_moment query; use it when you need the complete picture rather than a single answer (e.g. "walk through everything that happens in this video").
Requires a "video_id" from a prior understand_video or transcribe_video call.
| Name | Required | Description | Default |
|---|---|---|---|
| video_id | Yes | A video id previously returned by understand_video or transcribe_video. |
| Name | Required | Description |
|---|---|---|
| events | Yes | Every speech and visual event, merged and sorted chronologically. |
| video_id | Yes | |
| duration_seconds | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint false, so the safety profile is covered. The description adds meaningful behavioral context about the merged chronological view coming from a previously-processed video and the dependency on a prior understanding call.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is composed of three efficient sentences with no filler. The core return semantics are stated first, followed by the usage context and the required prerequisite.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter read-only tool with an output schema present, the description gives the key return semantics, the prerequisite, and the relationship to sibling tools. An agent has enough information to decide when to invoke it and what it will receive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already has 100% coverage for the single required video_id parameter, including the prerequisite that it was returned by understand_video or transcribe_video. The description adds little beyond what the schema already provides, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the exact resource and behavior: a full chronological, merged timeline of a previously-understood video, interleaving spoken segments and visual observations. It also distinguishes this tool from siblings by positioning it as the underlying structure that search_video and find_moment query.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says to use this tool when the agent needs the complete picture rather than a single answer, and connects it to the search siblings. It also states the prerequisite that video_id must come from a prior understand_video or transcribe_video call.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Searches across every information channel of a previously-understood video — spoken transcript, on-screen text, and visual context — for a word, phrase, name, or topic. Returns every match with its own timestamp and which modality it came from ("speech" vs. an on-screen/visual type), ranked by relevance.
Requires a "video_id" from a prior understand_video or transcribe_video call. Use this instead of re-reading a whole transcript when you already know what you're looking for — e.g. "pricing", "$49", "the dashboard", "AI agents".
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | What to search for — a word, phrase, name, number, or topic. | |
| video_id | Yes | A video id previously returned by understand_video or transcribe_video. |
| Name | Required | Description |
|---|---|---|
| query | Yes | |
| results | Yes | Ranked matches across speech AND on-screen/visual content, most relevant first. |
| video_id | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses that the operation is read-only (consistent with annotations), returns every match with timestamp and modality, and ranks by relevance. It also notes the dependency on prior video understanding, which annotations do not cover. This adds meaningful behavioral context beyond the readOnlyHint and idempotentHint annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two paragraphs: the first clearly defines function and output; the second gives usage guidance with examples. It is front-loaded with the core purpose, and every sentence adds value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the presence of an output schema (not shown) and the straightforward nature of a read-only search, the description covers all necessary context: what it searches, what it returns, prerequisites, and when to use it. No critical information appears missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema descriptions already cover both parameters (100% coverage), so baseline is 3. The description adds extra meaning: video_id must come from specific prior calls (understand_video or transcribe_video), and query can be a word, phrase, name, number, or topic. This exceeds the level of the input schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('searches') and resource ('every information channel of a previously-understood video, including spoken transcript, on-screen text, and visual context'), and specifies the return of matches with timestamps and modality. It clearly distinguishes from siblings like transcribe_video and understand_video by focusing on search rather than extraction or summarization.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use the tool ('when you already know what you're looking for'), the prerequisite ('video_id from a prior understand_video or transcribe_video call'), and names the alternative behavior it replaces ('re-reading a whole transcript'). This gives clear routing guidance relative to siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Retrieves a public Instagram Reel, TikTok video, or YouTube video/Short and returns an accurate, timestamped transcript of its spoken audio — and, when visual analysis is enabled, meaningful visual information the video shows.
WHAT IT DOES
Downloads the video/audio behind a public URL and transcribes the spoken speech using an automatic speech recognition model.
Returns structured, timestamped segments plus a combined plain-text transcript, along with the detected spoken language and total duration.
When visual analysis is enabled server-side, also returns "visual" — a short list of meaningful visual observations (on-screen text/slides, charts, UI, important scene context) with their own timestamps. This is never a caption for every frame; it only includes what a reader would actually need to understand the video without watching it. Absent or empty "visual" does not mean nothing was shown — it means nothing met that bar (or visual analysis wasn't enabled).
WHAT IT DOES NOT DO
It does NOT work on private, login-gated, deleted, or otherwise inaccessible content, and it never attempts to bypass login, CAPTCHAs, or other access controls — such content returns a typed error instead.
It does NOT currently support any platform other than Instagram Reels, TikTok videos, and YouTube videos/Shorts (no X/Twitter, etc.), and no Instagram content types other than Reels (no photo posts, carousels, Stories, or IGTV).
Visual analysis, when it runs, does not describe trivial visual activity (a person moving, blinking, camera motion) — only information that materially helps understand the content.
SUPPORTED INPUT
Public Instagram Reel, TikTok video, or YouTube video/Short URLs, e.g.: https://www.instagram.com/reel/ABC123xyz/ https://www.tiktok.com/@user/video/123... https://www.youtube.com/shorts/ABC123xyz
Any other URL shape or domain returns an UNSUPPORTED_SOURCE error.
LIMITATIONS
Videos are subject to a configured maximum duration and file size; longer/larger videos are rejected rather than partially processed.
Background music, overlapping speakers, heavy accents, or very noisy audio can reduce accuracy; when the model itself is uncertain, "low_confidence" is set to true instead of guessing at unclear speech.
This is a best-effort automatic transcript, not a human-verified one. Visual observations, when present, are similarly best-effort — on-screen text is preserved exactly as read, never "corrected", and a low "confidence" means treat it as uncertain rather than fact.
Use this tool when you need the words spoken in a public video from one of the supported platforms, and optionally what it visually showed.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | A public video URL — Instagram Reel (e.g. "https://www.instagram.com/reel/ABC123xyz/"), TikTok (e.g. "https://www.tiktok.com/@user/video/123..."), or YouTube/Shorts (e.g. "https://www.youtube.com/shorts/ABC123xyz" or "https://youtu.be/ABC123xyz"). Must point to content that does not require login to view. | |
| includeVideoMap | No | Whether to also resolve "this"/"that"/pointing references to a specific visual entity (the Video Map). Only has an effect when the server has the Video Map enabled AND includeVisualObservation is not false — it enriches the same sampled frames, not a standalone stage. Defaults to true (matches existing behavior) — pass false to skip it. | |
| includeVisualObservation | No | Whether to also analyze visual content (on-screen text, slides, charts, meaningful scene context) alongside the spoken transcript. Only has an effect when the server has visual analysis enabled at all; otherwise silently ignored. Defaults to true (matches existing behavior) — pass false to skip it for this call even when the server supports it. |
| Name | Required | Description |
|---|---|---|
| map | No | Present only when the Video Map (VIDEO_MAP_ENABLED) resolved at least one "this"/"that"/pointing reference to a specific visual target. Use get_video_map for the full picture including unresolved/uncertain ones. |
| url | Yes | The canonical URL that was transcribed. |
| text | Yes | The full transcript as plain text, in order. |
| source | Yes | The platform the video was retrieved from. |
| visual | No | Meaningful visual information detected in the video — on-screen text, slides, charts, UI, or scene context necessary to understand what is being discussed. Present only when visual analysis is enabled and found something worth surfacing; absent doesn't mean nothing was shown, only that nothing met the bar. This is never a caption for every frame — expect a short list of high-value observations, not a play-by-play. |
| language | No | BCP-47-ish language code detected in the spoken audio (e.g. "en"), if it could be determined. |
| segments | Yes | Timestamped transcript segments, in chronological order. |
| video_id | No | Stable id for this video — pass this to search_video, find_moment, and get_video_timeline instead of re-submitting the URL. |
| low_confidence | Yes | True when parts of the audio were unclear, mostly music/silence, or otherwise low-confidence. When true, treat the transcript as best-effort rather than verbatim. |
| duration_seconds | Yes | Total duration of the video/audio in seconds. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is clear. The description adds substantial behavioral context beyond annotations: it explains that visual analysis is server-side gated, that 'visual' is not a frame-by-frame caption, that low_confidence is set rather than guessing, that on-screen text is preserved exactly as read, and that longer/larger videos are rejected rather than partially processed. This is rich, honest behavioral disclosure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured with clear section headers (WHAT IT DOES, WHAT IT DOES NOT DO, SUPPORTED INPUT, LIMITATIONS). The most important information is front-loaded in the first sentence. Every section earns its place, though the length is at the upper bound of what is reasonable for a tool description.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (3 parameters, output schema, multiple supported platforms, server-side feature gating), the description is remarkably complete. It covers supported inputs, unsupported inputs, error behavior, limitations, confidence semantics, and the meaning of absent 'visual' data. The output schema exists, so return values don't need to be re-explained. Nothing an agent needs to call this correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all three parameters well. The description adds value by explaining the semantics of the visual output ('visual' is only meaningful observations, not every frame) and by clarifying that includeVideoMap only enriches the same sampled frames. However, the description doesn't add much beyond the schema for the url parameter, which is already thoroughly described in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb ('Retrieves... and returns') and names the exact resources (public Instagram Reel, TikTok video, YouTube video/Short) and outputs (timestamped transcript, visual info). It clearly distinguishes itself from siblings like understand_video, search_video, and find_moment by focusing on transcription of spoken audio plus optional visual observations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use the tool ('when you need the words spoken in a public video... and optionally what it visually showed') and provides extensive what-it-does-not-do guidance, including unsupported platforms, private content, and unsupported URL shapes. It also names the error type (UNSUPPORTED_SOURCE) for invalid inputs, which helps an agent decide before calling.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Watches a public video end to end and returns a full multimodal understanding of it: the spoken transcript AND the meaningful visual information it shows (on-screen text, charts, products, UI, scene context) — always with visual analysis requested, unlike transcribe_video where it's opt-in.
Use this as the entry point for any question that isn't purely "what was said" — anything about what was shown, displayed, or visible. The returned "video_id" can then be passed to search_video (find every mention of a topic across both speech and on-screen content), find_moment (get the single best timestamped piece of evidence for a question), and get_video_timeline (the full chronological merge of both modalities) — without re-submitting the URL or re-processing the video.
WHAT IT DOES NOT DO
Same platform/access limitations as transcribe_video: no private/login-gated/deleted content, no platforms beyond Instagram Reels, TikTok, and YouTube videos/Shorts.
Does not itself answer free-form questions — call search_video or find_moment on the returned video_id for that.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | A public video URL — Instagram Reel, TikTok, or YouTube/Shorts. Must point to content that does not require login to view. |
| Name | Required | Description |
|---|---|---|
| map | No | Present only when the Video Map (VIDEO_MAP_ENABLED) resolved at least one "this"/"that"/pointing reference to a specific visual target. Use get_video_map for the full picture including unresolved/uncertain ones. |
| url | Yes | |
| text | Yes | |
| source | Yes | |
| visual | No | Visual observations detected — on-screen text, charts, products, scene context, etc. |
| language | No | |
| segments | Yes | |
| video_id | No | Stable id for this video — pass this to search_video, find_moment, and get_video_timeline. |
| low_confidence | Yes | |
| duration_seconds | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
While annotations already declare readOnly, idempotent, and openWorld hints, the description adds behavioral detail beyond those: it clarifies the output includes both transcript and visual analysis, explains that the returned video_id is reusable across other tools without reprocessing, and explicitly states it does not answer free-form questions. This contextualizes the tool's behavior without contradicting any annotation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is longer than average but well-structured: purpose and entry-point guidance are front-loaded, followed by downstream usage and a dedicated 'WHAT IT DOES NOT DO' section. Every sentence contributes useful information, though it could be tightened slightly without losing clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's multimodal complexity and integration with multiple siblings, the description is remarkably complete. It covers what it does, what it does not do, the limitations, and how the output (video_id) connects to other tools. Since an output schema exists, omitting return structure details is acceptable. Nothing an agent needs to decide when and how to call it is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already provides full coverage (100%) for the single 'url' parameter, describing it as a public video URL with platform restrictions and login requirements. The description reiterates these constraints in the 'DOES NOT DO' section but does not add new parameter-level meaning beyond what the schema already conveys, keeping it at the baseline for high coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('watches') and resource ('public video') and clearly defines the output as both transcript and visual information. It explicitly differentiates itself from transcribe_video by noting that visual analysis is always included versus opt-in for its sibling, making its purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit entry-point guidance ('Use this as the entry point for any question that isn't purely what was said'), states what it does NOT do (does not answer free-form questions), and names the exact sibling tools to use for downstream tasks (search_video, find_moment, get_video_timeline). It also lists platform and access limitations, leaving no inference required.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0find_momentget_video_entityget_video_evidenceget_video_mapget_video_timelinesearch_videotranscribe_videounderstand_videoScored across 8 tools
Each tool has a clearly distinct role: transcribe_video for audio-only transcription, understand_video for full multimodal processing, and the remaining tools operate on the returned video_id for search, evidence, timeline, or semantic mapping. Even the overlapping transcribe/understand pair is explicitly differentiated by use case, and search_video vs find_moment are separated by exhaustive vs. single-best results.
All tool names follow a consistent snake_case verb_noun pattern (transcribe_video, understand_video, search_video, find_moment, get_video_timeline, get_video_map, get_video_entity, get_video_evidence). The pattern is uniform and predictable, making it easy for an agent to infer function from the name alone.
Eight tools is well within the ideal 3–15 range and feels well-scoped for a video understanding server. Each tool has a specific place in the pipeline (ingest, search, timeline, semantic map, evidence lookup) without redundancy or bloat.
The tool surface covers the full workflow: ingest a video (transcribe_video/understand_video), explore it (search_video, find_moment, get_video_timeline), and dig into semantic relationships (get_video_map, get_video_entity, get_video_evidence). The design anticipates the questions agents will ask and provides no dead ends—every returned video_id can be used by downstream tools.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Social media analytics, video analysis, and competitor intel for any MCP-compatible AI agent.
Any social-video URL → transcript, metadata, frames, OCR, summary, search, Q&A. MCP server + x402.
Build, run, schedule, and publish AI video pipelines to YouTube and TikTok from any MCP client.
OCR, transcription, file extraction, and image generation for AI agents via MCP.
