Skip to main content
Glama
Sign In
moreel

moreel

by medaharrat
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

CI License: MIT Node.js >= 22 TypeScript MCP compatible

Most video-to-transcript tools stop at the audio. Moreel treats the video itself as the source of truth: it transcribes what was said, reads what was shown (on-screen text, slides, charts, products), and resolves what a vague "this one" or a pointing gesture actually meant — then makes all of it searchable and jumpable, down to the exact second.

It's built to be used three ways:

  • As a web app — paste a URL, get video + transcript side by side, with in-transcript search and a "what did I miss?" panel for anything shown but never said out loud.

  • As an HTTP API — the same pipeline, for your own backend/integration.

  • As an MCP server — a set of tools (understand_video, search_video, find_moment, get_video_map, ...) so an AI agent can query a video without ever "watching" it or scrubbing through a transcript by hand.

1. What it does

  • Transcription — accurate, timestamped speech-to-text (OpenAI Whisper), adaptively re-chunked from word-level timestamps so a segment reflects how the person actually spoke, not an arbitrary decode boundary.

  • Visual understanding (opt-in) — bounded frame sampling + a vision model surfaces on-screen text, slides, charts, products, and scene context, kept as a separate layer from the transcript, never merged into it.

  • The Video Map (opt-in, layered on visual understanding) — resolves "this"/"that"/"this one" and pointing/showing/holding gestures to a specific visual entity, with a confidence level and never a fabricated target when the evidence is weak.

  • Unified search — one query surfaces matches across speech, on-screen text, and resolved references, each labeled by source and one click from seeking the player. Lexical by default; opt-in embeddings add semantic matching (a query for "cost" also finds on-screen text that says "pricing").

  • "What did I miss?" — surfaces exactly what was visible but never said, classified (on-screen text, visual context, an unspoken visual reference, ...), each anchored to a timestamp — never a generic summary.

Supported sources: public Instagram Reels, TikTok videos, and YouTube videos/Shorts. Private, login-gated, or otherwise access-controlled content is never attempted — it fails with a typed error by design.

Related MCP server: ttagent

2. Installation

Requirements:

  • Node.js >= 22

  • yt-dlp on PATH (or set YTDLP_PATH)

  • ffmpeg on PATH (or set FFMPEG_PATH)

  • An OpenAI API key (transcription; also vision/Video Map/embeddings if you enable those)

git clone https://github.com/medaharrat/moreel.git
cd moreel
npm install
cp .env.example .env
# edit .env and set OPENAI_API_KEY

cd web && npm install && cd ..

3. Running it

Quickest path — Docker Compose (API + web UI + Postgres + Redis):

export OPENAI_API_KEY=sk-...
# optional: export VISION_ENABLED=true VIDEO_MAP_ENABLED=true
docker compose -f docker-compose.prod.yml up --build

Web UI at http://localhost:3001, API at http://localhost:8080.

From source, without Docker:

npm run build && npm run migrate:up   # Postgres schema (optional — see below)

npm run dev:http     # HTTP API + web-facing endpoints, from source
cd web && npm run dev  # web UI (separate terminal)

npm run dev           # or: the MCP server over stdio, for an agent client

Postgres/Redis are optional in development — without DATABASE_URL, video records fall back to an in-process cache; without REDIS_URL, rate limiting/dedup fall back to in-memory, single-replica-only behavior.

4. Configuration

All configuration is via environment variables — see .env.example for the full, documented list. The important ones:

Variable

Default

Purpose

OPENAI_API_KEY

required

Transcription (and vision/Video Map/embeddings, if enabled)

VISION_ENABLED

false

On-screen text/scene understanding alongside the transcript

VIDEO_MAP_ENABLED

false

Resolve "this"/"that"/pointing references (requires VISION_ENABLED)

SEARCH_EMBEDDINGS_ENABLED

false

Semantic matching on top of lexical search

MAX_VIDEO_SIZE_MB

100

Reject media larger than this

MAX_VIDEO_DURATION_SECONDS

600

Reject media longer than this

MAX_CONCURRENT_REQUESTS

4

Bounded concurrency; excess fails fast with RATE_LIMITED

CACHE_ENABLED

true

Cache processed videos (in-memory, or Postgres/Redis if configured)

DATABASE_URL

(none)

Enables durable, cross-replica video storage + accounts/API keys

REDIS_URL

(none)

Enables distributed rate limiting/dedup

LOG_LEVEL

info

Structured log verbosity

Never hardcode secrets — always via environment/.env.

5. MCP tools

Point any MCP client at node dist/mcp/server.js (stdio transport):

{
  "mcpServers": {
    "moreel": {
      "command": "node",
      "args": ["/absolute/path/to/moreel/dist/mcp/server.js"],
      "env": { "OPENAI_API_KEY": "sk-..." }
    }
  }
}

Tool

Does

transcribe_video

Speech only — timestamped transcript, opt-in visual observations

understand_video

Speech + visual understanding, always on — entry point for most agent use

search_video

Query across speech, on-screen text, and resolved references

find_moment

The single best timestamped answer to a specific question

get_video_timeline

The full chronological merge of every modality

get_video_map

Entities, interactions, and references the Video Map resolved

get_video_entity

Everything tied to one specific entity (every mention, every moment)

get_video_evidence

The underlying evidence behind one specific map fact, by id

search_video/find_moment/get_video_map operate on a video_id returned by a prior transcribe_video/understand_video call — an agent watches a video once, then queries it repeatedly without re-processing.

Every response is structured JSON — timestamps, confidence, and observed/inferred/uncertain evidence levels — never prose an agent has to parse.

6. Architecture

INGEST          src/providers/     VideoProvider interface; Instagram/TikTok/YouTube
                                     (yt-dlp), each behind a circuit breaker + rate limit
      ↓
MEDIA            src/media/         SSRF-guarded download, audio extraction, bounded
                                     frame sampling (periodic + scene-change)
      ↓
TRANSCRIPTION    src/transcription/ Transcriber interface; OpenAI Whisper; word-timestamp
                                     re-chunking; hallucination/confidence normalization
      ↓
VISION           src/vision/        VisionProvider (on-screen text/scene) +
                                     VideoInteractionAnalyzer (the Video Map), opt-in
      ↓
APPLICATION      src/app/           Timeline merge, lexical+semantic search, "what did I
                                     miss", video-map resolution — orchestration only,
                                     never provider-specific
      ↓
TRANSPORTS       src/mcp/           MCP tools (stdio)
                 src/http/          Web UI's API + REST-ish JSON routes

src/domain/ holds the shared types (Transcript, VisualObservation, VideoRecord, the Video Map's Interaction/LinguisticReference) and a typed MoreelError taxonomy — nothing above it knows which platform or which model vendor produced the data. Adding a platform means one more VideoProvider; adding a model vendor means one more Transcriber/ VisionProvider — nothing else changes.

Cross-cutting concerns live alongside, not inside, the pipeline: src/cache/, src/observability/, src/usage/, src/config/, src/util/.

Security posture

  • URL validation & SSRF protection — every input URL is parsed and rejected if malformed, non-http(s), or pointing at a private/loopback/ link-local IP (including the cloud-metadata address); redirects are re-validated the same way (src/media/downloader/ssrf.ts).

  • No shell interpolation — yt-dlp/ffmpeg run via execFile with argument arrays, never a shell.

  • No path traversal — downloaded files live under a server-generated, randomly named per-request temp directory, always cleaned up in a finally block, regardless of success or failure.

  • Bounded everything — size/duration caps enforced while streaming, hard timeouts on every network/subprocess call, bounded concurrency that fails fast (RATE_LIMITED) instead of queuing unboundedly.

  • No secret leakage — typed errors carry a client-safe message; stack traces, file paths, and provider internals are logged server-side only.

  • Bounded-retention storage, not an archive — processed videos (when persisted to Postgres/Redis) carry an explicit TTL and are treated as a cache, matching the same minimal-retention principle as the in-process cache. See docs/privacy.md.

7. Development

npm run dev          # MCP server from source (tsx), no build step
npm run dev:http      # HTTP API from source
npm run typecheck    # tsc --noEmit
npm run lint         # eslint
npm run format       # prettier --write

8. Testing

npm test                 # everything
npm run test:unit        # provider selection, media limits, normalization,
                          # timeline/search/video-map resolution, config, ...
npm run test:integration # full pipeline, real Postgres video-store round-trips
npm run test:protocol    # real MCP Client <-> Server over an in-memory transport
npm run test:regression  # transcription quality regression corpus

All network access and subprocess execution (yt-dlp, ffmpeg, the OpenAI API) are mocked at their interface boundaries in unit/protocol tests — the integration suite's Postgres-backed tests are the exception, and skip automatically when DATABASE_URL isn't reachable.

9. Limitations

  • Public content only, by design — Moreel never bypasses login, CAPTCHAs, or other access controls.

  • Automatic transcription/visual analysis is best-effort, not human-verified — surfaced via low_confidence/evidenceLevel rather than silently guessed.

  • No speaker diarization — multi-speaker audio transcribes in order without speaker labels (Whisper doesn't expose this).

  • The Video Map doesn't track entity identity via visual similarity across a whole video — it relies on the model reusing a consistent label for a recurring entity within one analysis pass.

  • Video Map events don't yet carry their own evidence frame thumbnail (timestamp + confidence still make them fully traceable via the player).

10. Contributing

Issues and PRs are welcome. npm run typecheck && npm run lint && npm test should pass before opening one — CI runs the same checks (plus integration/protocol/regression suites against real Postgres+Redis service containers) on every PR.

11. License

MIT — see LICENSE.

Available Tools

8 tools
find_momentFind MomentA
Read-onlyIdempotent

Returns the single best timestamped piece of evidence in a previously-understood video for a specific question — precise, verifiable, and anchored to one moment, unlike search_video which returns every match.

Requires a "video_id" from a prior understand_video or transcribe_video call. Use this when you need one authoritative answer with proof (e.g. "does the creator show pricing anywhere?", "what is he pointing at when he says 'this one'?") rather than a list of every mention. When the best match resolves to a specific visual entity (a pointing gesture, a "this"/"that" reference), the "interaction" field gives the structured target — absent when evidence was too weak to confidently resolve one, never a fabricated guess.

Inputs: query (required), video_id (required). Returns: text, found, query, answer, +7 more

get_video_entityGet Video EntityA
Read-onlyIdempotent

Returns everything the Video Map knows about one specific entity (a person/product/object) from get_video_map — every interaction it was involved in and every linguistic reference resolved to it, in chronological order.

Use this after get_video_map or search_video/find_moment surfaces an entity id, to see its full history across the video (e.g. "every time the product was pointed at or referred to") rather than one isolated moment.

Inputs: video_id (required), entity_id (required). Returns: type, found, label, video_id, +7 more

get_video_evidenceGet Video EvidenceA
Read-onlyIdempotent

Looks up the full evidence behind a single Video Map fact by id — never invents detail beyond what was actually recorded (see get_video_map, get_video_entity). Use this to double-check a specific claim before treating it as authoritative, e.g. confirming a reference's evidence_level before repeating its resolved target as fact.

Inputs: video_id (required), evidence_id (required). Returns: kind, found, detail, modality, +5 more

get_video_mapGet Video MapA
Read-onlyIdempotent

Returns the Video Map: the semantic layer connecting what was SAID to what was VISIBLE and what the person DID — entities (people/products/objects that recur across the video), interactions (points_at, shows, holds, etc.), and references (what "this"/"that one" actually meant).

This is what resolves "speech → reference → visual target" — a transcript alone cannot tell you what "this one" refers to; this can, when the visual evidence was strong enough. Every interaction/reference has an "evidence_level" (observed/inferred/uncertain) and a "confidence" — absent "target_entity_id" means the evidence was too weak to confidently resolve a target, which is the CORRECT and expected answer in that case, not a failure.

Requires a "video_id" from a prior understand_video call with visual analysis enabled. Returns empty arrays (not an error) when the Video Map feature wasn't enabled or found nothing — use search_video/find_moment for the underlying speech/visual search either way.

Inputs: video_id (required). Returns: scenes, entities, video_id, references, +1 more

get_video_timelineGet Video TimelineA
Read-onlyIdempotent

Returns the full chronological, merged timeline of a previously-understood video — every spoken segment interleaved with every visual observation, in time order. This is the underlying structure search_video and find_moment query; use it when you need the complete picture rather than a single answer (e.g. "walk through everything that happens in this video").

Requires a "video_id" from a prior understand_video or transcribe_video call.

Inputs: video_id (required). Returns: events, video_id, duration_seconds

search_videoSearch VideoA
Read-onlyIdempotent

Searches across every information channel of a previously-understood video — spoken transcript, on-screen text, and visual context — for a word, phrase, name, or topic. Returns every match with its own timestamp and which modality it came from ("speech" vs. an on-screen/visual type), ranked by relevance.

Requires a "video_id" from a prior understand_video or transcribe_video call. Use this instead of re-reading a whole transcript when you already know what you're looking for — e.g. "pricing", "$49", "the dashboard", "AI agents".

Inputs: query (required), video_id (required). Returns: query, results, video_id

transcribe_videoTranscribe VideoA
Read-onlyIdempotent

Retrieves a public Instagram Reel, TikTok video, or YouTube video/Short and returns an accurate, timestamped transcript of its spoken audio — and, when visual analysis is enabled, meaningful visual information the video shows.

WHAT IT DOES

  • Downloads the video/audio behind a public URL and transcribes the spoken speech using an automatic speech recognition model.

  • Returns structured, timestamped segments plus a combined plain-text transcript, along with the detected spoken language and total duration.

  • When visual analysis is enabled server-side, also returns "visual" — a short list of meaningful visual observations (on-screen text/slides, charts, UI, important scene context) with their own timestamps. This is never a caption for every frame; it only includes what a reader would actually need to understand the video without watching it. Absent or empty "visual" does not mean nothing was shown — it means nothing met that bar (or visual analysis wasn't enabled).

WHAT IT DOES NOT DO

  • It does NOT work on private, login-gated, deleted, or otherwise inaccessible content, and it never attempts to bypass login, CAPTCHAs, or other access controls — such content returns a typed error instead.

  • It does NOT currently support any platform other than Instagram Reels, TikTok videos, and YouTube videos/Shorts (no X/Twitter, etc.), and no Instagram content types other than Reels (no photo posts, carousels, Stories, or IGTV).

  • Visual analysis, when it runs, does not describe trivial visual activity (a person moving, blinking, camera motion) — only information that materially helps understand the content.

SUPPORTED INPUT

LIMITATIONS

  • Videos are subject to a configured maximum duration and file size; longer/larger videos are rejected rather than partially processed.

  • Background music, overlapping speakers, heavy accents, or very noisy audio can reduce accuracy; when the model itself is uncertain, "low_confidence" is set to true instead of guessing at unclear speech.

  • This is a best-effort automatic transcript, not a human-verified one. Visual observations, when present, are similarly best-effort — on-screen text is preserved exactly as read, never "corrected", and a low "confidence" means treat it as uncertain rather than fact.

Use this tool when you need the words spoken in a public video from one of the supported platforms, and optionally what it visually showed.

Inputs: url (required), includeVideoMap, includeVisualObservation. Returns: map, url, text, source, +6 more

understand_videoUnderstand VideoA
Read-onlyIdempotent

Watches a public video end to end and returns a full multimodal understanding of it: the spoken transcript AND the meaningful visual information it shows (on-screen text, charts, products, UI, scene context) — always with visual analysis requested, unlike transcribe_video where it's opt-in.

Use this as the entry point for any question that isn't purely "what was said" — anything about what was shown, displayed, or visible. The returned "video_id" can then be passed to search_video (find every mention of a topic across both speech and on-screen content), find_moment (get the single best timestamped piece of evidence for a question), and get_video_timeline (the full chronological merge of both modalities) — without re-submitting the URL or re-processing the video.

WHAT IT DOES NOT DO

  • Same platform/access limitations as transcribe_video: no private/login-gated/deleted content, no platforms beyond Instagram Reels, TikTok, and YouTube videos/Shorts.

  • Does not itself answer free-form questions — call search_video or find_moment on the returned video_id for that.

Inputs: url (required). Returns: map, url, text, source, +6 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Deploy Server
    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    TDQS

    A4.7/5.0

    Scored across 8 tools

    Disambiguation5/5

    Each tool has a clearly distinct role: transcribe_video for audio-only transcription, understand_video for full multimodal processing, and the remaining tools operate on the returned video_id for search, evidence, timeline, or semantic mapping. Even the overlapping transcribe/understand pair is explicitly differentiated by use case, and search_video vs find_moment are separated by exhaustive vs. single-best results.

    Naming Consistency5/5

    All tool names follow a consistent snake_case verb_noun pattern (transcribe_video, understand_video, search_video, find_moment, get_video_timeline, get_video_map, get_video_entity, get_video_evidence). The pattern is uniform and predictable, making it easy for an agent to infer function from the name alone.

    Tool Count5/5

    Eight tools is well within the ideal 3–15 range and feels well-scoped for a video understanding server. Each tool has a specific place in the pipeline (ingest, search, timeline, semantic map, evidence lookup) without redundancy or bloat.

    Completeness5/5

    The tool surface covers the full workflow: ingest a video (transcribe_video/understand_video), explore it (search_video, find_moment, get_video_timeline), and dig into semantic relationships (get_video_map, get_video_entity, get_video_evidence). The design anticipates the questions agents will ask and provides no dead ends—every returned video_id can be used by downstream tools.

    Maintenance

    ActivityStale
    ResponsivenessNo recent community issues

    Resources

    Related MCP Connectors

    Related MCP Servers

    • Moonscroll MCP Server

      Moonscroll MCP Serverofficial

      thefedoration
      A
      license
      Not graded
      quality
      C
      maintenance
      Enables agents to research and analyze public short-form video content on Instagram Reels, TikTok, and YouTube Shorts for trend spotting, creator research, competitive briefs, and content ideation.
      1
      MIT

    • ttagent

      Bilal140202
      A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to extract TikTok video and soundtrack media, metadata, covers, and manifests from public URLs or IDs through verified downloads and schema access via MCP tools.
      4
      MIT

    • mcptube-vision

      OzlemHill118
      F
      license
      Not graded
      quality
      A
      maintenance
      Enables MCP clients such as Claude and Copilot to turn YouTube videos into a persistent, compounding wiki by extracting knowledge from transcripts and scene-change vision analysis. Lets clients run agentic search over that compiled knowledge, answering questions with source citations and awareness of what the wiki does not yet cover.
      -

    • igsight-mcp

      vinihuge
      A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to retrieve and analyze Instagram posts and profiles, including captions, media, video frames, transcripts, and engagement metrics, through MCP tools.
      67 PyPI
      MIT