video-research-mcp
This server exposes 120 MCP tools for video, audio, image, document, web/academic research, and saved-knowledge workflows.
Analyze videos from YouTube or local files, including custom instructions, structured schemas, batches, multi-turn sessions, and windowed long-video analysis.
Inspect and export media: metadata, frames, contact sheets, filmstrips, scenes, keyframes, storyboards, clips, audio clips, crops, and deduplicated frames/audio.
Transcribe audio, analyze DSP/music, compare audio, and export captions/transcripts with provenance.
Process images and vision: read, edit, crop, OCR, segment, vision chat, grounding, manifest verification, and reverse image search.
Analyze documents and content: extract structured data, compare files, batch analyze, and ingest PDFs, DOCX, Markdown, HTML, text, or audio.
Run research workflows: deep synthesis, research planning, evidence assessment, document-grounded research, web Deep Research, academic paper search/details/citations/recommendations/authors, and source-preserving execution.
Search and extract web content, use optional search providers, and inspect provider capabilities.
Store and retrieve knowledge in Weaviate-style collections: search, related objects, stats, fetch, ingest, ask, schema, and deprecated query.
Manage infrastructure: runtime configuration, cache, job status/cancel, session memory, source ingestion, and media asset storage.
Call optional providers such as TwelveLabs, MHS simulated devices, and generic text generation backends.
Supports direct analysis of academic papers from arXiv to extract entities, relationships, and key arguments.
Integrates with ElevenLabs to provide a media production toolkit for generating audio and cinematic video content.
Uses FFmpeg for extracting frames from local video files to facilitate visual analysis and timestamped screenshotting.
Provides web search capabilities with source citations through Gemini's search grounding features.
Serves as the core engine for video understanding, deep research, context caching, and document synthesis.
Provides tools for querying, debugging, and evaluating MLflow traces generated during research and analysis workflows.
Enables comprehensive analysis of YouTube content, including metadata extraction, comment sentiment analysis, and playlist processing.
video-research-mcp
Ask questions across recordings and documents, then inspect the source frames, transcripts and clips behind the answer. The server exposes 120 MCP tools for video, audio and image operations, document analysis, web and academic research, and saved knowledge. Agent workflows connect those tools into a research or production task.
Use it in Codex, Claude Code, or any client that supports stdio Model Context Protocol. Gemini handles the main analysis and research routes; local tools inspect and transform source files. Optional providers and runtimes extend that core.
Open the interactive user guide (Dutch): from your problem to an approach. Search the function map or choose a task to see its tools, prerequisites and steps. The English user documentation also covers setup problems and examples. Describe your problem in your MCP client in ordinary language to begin.
Release candidate: 0.8.0-rc.6 (PyPI 0.8.0rc6). The links above
identify the exact packages and immutable source tag. Check the release notes
and each route's prerequisites before installing. The npm prerelease channel
is next. Local-model qualification is outside this API-first release.
Stable 0.7.1 predates the native Codex plugin and expanded media surface.
From source material to a useful result
Work | Tools to start with | What to inspect |
Question a recording |
| Source timestamps, the answer and supporting material |
Analyze a long recording in stages |
| Requested intervals, dry-run plan, execution limits and retained outcomes |
Prepare audio or transcripts |
| Caption or model provenance, source timing and exported audio |
Inspect images or export media |
| Source bytes, selected region or interval, output files and manifests |
Compare documents or investigate a topic |
| Citations, contradictions, missing evidence and job status |
Generate images or short video | Separate explainer image and video APIs | Pinned sources, a current operator quote, explicit operations and saved decoded files; provider quality remains unqualified |
Search previous work |
| Stored records and retrieval provenance; requires Weaviate |
For example, ask your client:
Compare ~/recordings/design-review.mp4 with ~/docs/requirements.pdf.
Identify decisions that change a requirement. Cite recording timestamps and
relevant document pages, distinguish explicit decisions from interpretation,
and flag anything the sources do not establish.Workflows combine tools; individual MCP calls return their own results. Source hashes identify bytes, and export manifests retain how an artifact was produced. Model timestamps, citations and interpretations still need checking against the original material. Sampling a recording does not establish complete coverage.
Related MCP server: media-mcp
Install
You need Python 3.11+, uv
with uvx on your client's PATH, and a Gemini API key
for Gemini requests. Plugin installation also needs Node.js 22+ and npm.
Install FFmpeg/FFprobe for local media inspection, frame extraction and clip
exports. Other operations may need an optional dependency or configured backend.
By default, the server reads process environment variables first, then
~/.config/video-research-mcp/.env, then defaults. Project installations can select
a separate credential file. Add your key to the shared file for the routes below,
keeping any existing settings:
GEMINI_API_KEY=your-gemini-api-keyKeep the file private (chmod 600 ~/.config/video-research-mcp/.env). Codex plugin
servers receive a filtered environment, so this file is the reliable credential
route there. Provider analysis sends source material to the configured service
and can incur usage charges. A selected analysis interval does not necessarily
limit the uploaded file.
Codex: native plugin
The plugin supplies 26 skills and the version-pinned research server. Historical RC3 observation: on 2026-10-06, the selected Codex 0.160.1 RC3 installation and fresh-session restart loaded the research tools and packaged skills. This is historical RC3 evidence. Follow the verification steps below for the version you install. The native route does not run the Claude workflow installer.
For a new installation, save this catalog as
~/.local/share/video-research-mcp/.agents/plugins/marketplace.json, creating its
parent directories if needed:
{
"name": "video-research",
"interface": { "displayName": "Video Research" },
"plugins": [
{
"name": "video-research",
"source": {
"source": "npm",
"package": "video-research-mcp",
"version": "0.8.0-rc.6"
},
"policy": { "installation": "AVAILABLE", "authentication": "ON_INSTALL" },
"category": "Productivity"
}
]
}codex plugin marketplace add ~/.local/share/video-research-mcp
codex plugin add video-research@video-research
codex plugin list --jsonStart a fresh Codex session. Check that video-research@video-research is enabled,
its source is npm at 0.8.0-rc.6, and the server's tools are available.
For an existing installation, use the
native plugin and migration guide.
Back up local plugin edits before reinstalling: Codex replaces its managed cache.
A manual mcp_servers.video-research entry can hide the plugin server. In the
historical RC1 check on Codex 0.160.0, this also occurred when the entry was
disabled; the guide describes the specific configuration to remove.
Claude Code: workflow installer
npx video-research-mcp@0.8.0-rc.6 --globalThis installs slash commands, skills and agents into ~/.claude/, registers the
research server in ~/.claude.json, and creates a credential template if needed.
Set the key, restart Claude Code and inspect /mcp. Use /gr:advisor to select a
workflow, or /gr:video, /gr:research and /gr:analyze to start directly.
Use --local for project scope and set the key in
./.config/video-research-mcp/.env; this replaces the shared credential-file route.
Use --global --check to inspect the global installation. Updates preserve modified
workflow files and custom configuration. See
installer options and recovery
for scope, checkpoints and rollback.
Other MCP clients: server only
Use your client's stdio registration format. A typical JSON entry is:
{
"mcpServers": {
"video-research": {
"command": "uvx",
"args": ["video-research-mcp==0.8.0-rc.6"]
}
}
}This connects the same tools; agent workflows are installed separately. Python
package filenames use the normalized spelling 0.8.0rc6.
Verify your first connection
Ask the client to call infra_configure with no arguments. It returns the
running configuration without making an analysis request. Then try a tool on
material you can inspect yourself. A successful connection verifies setup;
checking the returned evidence verifies the particular result.
Configure only what the task needs
Setting or addition | Use it for |
| Select supported primary and auxiliary models |
| YouTube metadata, comments and playlists; otherwise uses the Gemini key |
| Persist video sessions in SQLite; unset means in-memory sessions |
| Store and retrieve research across sessions |
| Trace tool execution; |
| Restrict supported local-file operations to a directory |
Research works without Weaviate. When storage is configured, write-through errors
are non-fatal; verify the stored record when persistence matters. The
knowledge-store guide
covers embeddings, collections and optional query dependencies. For a manually
configured server, retain the version pin when adding extras, for example
uvx 'video-research-mcp[tracing,agents]==0.8.0-rc.6'. The full runtime configuration lives in
ServerConfig.
Image edits, OCR, speech inference, Blender, FreeCAD and local model services have additional runtime or backend requirements. Installing the plugin does not install those runtimes, model weights or provider accounts. Inspect the relevant skill and its prerequisites before using an optional integration.
For video production, the separate explainer companion exposes 39 tools, including direct HTTP image and text/frame-video generation and an optional external CLI/rendering pipeline; the scene-agent companion exposes two tools for scene-code generation. Neither is required for research. Production skills also provide workflows for narration, image generation, clip generation and assembly.
Generate an image or a short video
Connect the explainer companion separately. Its RC4 candidate (0.2.2rc4 on PyPI)
adds direct HTTP routes for text-to-image, image editing, image translation and
video from text, a first frame, or first plus last frames. These routes need the
companion's generation extra, an existing project with pinned script/scene
files, explicit regional DashScope configuration and current pinned price/access
and quote declarations. Video output checks also need FFmpeg/ffprobe.
They run without the upstream video_explainer CLI.
Follow the image lifecycle
or video lifecycle.
Obtain actual human authority for source submission and spending before setting
spend_authorized=true; that field records an input acknowledgement and does not
authenticate anyone or impose a provider billing cap. Keep the returned job ID.
Use explainer_image_generation_finalize for captured synchronous images,
explainer_image_generation_poll for translation and explicit image recovery,
or explainer_generation_poll for video, each with a bounded authorized
operation. Core job_status rejects both generated-media job kinds.
Saved hashes and complete decode checks qualify the selected file bytes. Live provider access, image identity/style fidelity, creative quality and full security review remain open. S2V/HappyHorse successor modes are optional and await separate source acceptance; the disabled raw Qwen process integration is a separate route. Stock search, local ASR intake timing and the retained renderer timing failure remain limits.
Inspect and extend
Tool implementations: exact parameters and behavior. Your client's discovered MCP schemas describe the running version.
Windowed video analysis: dry-run budgets, continuation, upload limits and retained partial results.
Contributing: source setup and checks.
Security policy: local access, provider boundaries and reporting.
Changelog: changes in this candidate.
The selected RC3 native check covered the installed baseline and restart, with 124 cache files. Fresh npm network acquisition versus reuse of the same-version cache remains unknown. Earlier RC1 native and RC2 archive/installer checks remain historical results. Tool discovery does not qualify every optional runtime; live-provider quality and complete comparative acceptance remain open.
Created by Fausto Albers · Wonder Why. Built with Google Gemini, FastMCP and Pydantic, with optional knowledge and tracing integrations. Project code is MIT licensed; third-party notices cover bundled components with their own terms.
Available Tools
90 toolsaudio_clip_exportAudio Clip ExportA
Export a whole or one-sided audio selection to a unique private 16kHz mono WAV.
Originals and prior exports remain unchanged. Caller-selected output paths and overwriting are rejected. Use image_manifest_read for verified restart.
Inputs: request (required). Returns: object
audio_deduplicateAudio DeduplicateA
Compare local audio candidates while retaining silence, errors and uncertainty.
Requires the optional audio extra and local FFmpeg. Similarity does not establish equal speech, speaker identity or permission to delete evidence.
Inputs: request (required). Returns: object
audio_dsp_analyzeAudio Dsp AnalyzeC
Measure audio or invoke one separately installed operator-pinned native DSP workflow.
Local analysis preserves stereo, finite silence/abstentions and absolute clipping intervals. A/B uses reference-minus-primary differences with units. Optional Juzzy modes preserve harmonic/rhythm/masking/section observations; Ferrous receives an explicit mono derivation and retains heuristic quality, content and fingerprint output. Standards and semantic claims are unverified. Reusing an exact job_id reads its attested artifacts without launching again. Use job_status for durable status, including interrupted/failed work.
Inputs: request (required), include_image. Returns: object
audio_transcribeAudio TranscribeA
Extract exact caption intervals or infer speech through an explicit backend.
Supplied cue and word times remain source assertions. Model-derived words and speaker labels remain inference. Caption selection precedes ASR; local-only selection forbids cloud requests. Readback requires the retained receipt digest and exact source identity. No local model is installed.
Inputs: request (required). Returns: object
content_analyzeContent AnalyzeARead-only
Analyze content (file, URL, or text) with any instruction.
Provide exactly one of file_path, url, or text. Uses Gemini's structured output for reliable JSON responses. Pass a custom output_schema to control the response shape, or use the default ContentResult schema.
Inputs: url, text, file_path, instruction, +2 more. Returns: object
content_batch_analyzeContent Batch AnalyzeARead-only
Analyze multiple content files from a directory or explicit file list.
Rejects discovery over 5,000 entry visits and compare payloads over DOC_MAX_DOWNLOAD_BYTES. Supports two modes: 'compare' sends files in a single call for cross-document analysis, 'individual' analyzes each file separately with bounded concurrency (3 parallel calls).
Inputs: mode, directory, max_files, file_paths, +4 more. Returns: object
content_extractContent ExtractARead-only
Extract structured data from content using a JSON Schema.
Uses Gemini's response_json_schema for guaranteed structured output.
Inputs: schema (required), content (required). Returns: object
image_cropImage CropA
Crop local PNG pixels without model inference or provider uploads.
Requires an independently installed FFmpeg and respects LOCAL_FILE_ACCESS_ROOT. A source hash binds the crop to bytes; it does not establish factual support or whether the input was synthetic. Large outputs retain metadata/path only.
Inputs: crop_box (required), file_path (required), output_path (required), include_image. Returns: object
image_editImage EditA
Edit a source image or decoded video frame and persist its exact manifest.
Coordinates use the oriented source grid before crop and resize. Requires the optional images extra; frame selection also requires local FFmpeg. Source-derived edits and annotations retain explicit provenance.
Inputs: request (required), include_image. Returns: object
image_manifest_readImage Manifest ReadARead-onlyIdempotent
Read an exact manifest after restart and verify its source and artifact bytes.
Inputs: manifest_path (required), expected_sha256 (required), include_image. Returns: object
image_ocrImage OcrA
Observe text and geometry using an explicitly selected local OCR engine.
Source OCR and detected geometry do not establish semantic table structure or factual support. Missing local runtimes return an explicit limitation.
Inputs: request (required). Returns: object
image_readImage ReadB
Return a bounded local still image as inspectable PNG and exact source metadata.
Animated inputs expose one still with an explicit limitation. Native output may be processed by the host's model; this server invokes no model or upload.
Inputs: file_path (required), max_pixels, include_image. Returns: object
image_segmentImage SegmentA
Plan or submit one image and preserve actual mask and overlay artifacts.
Scores and masks are unverified proposals. Service selection and checkpoint declarations do not attest a loaded model or establish redistribution rights.
Inputs: request (required), include_image. Returns: object
infra_cacheInfra CacheADestructiveIdempotent
Manage the analysis cache — stats, list, clear, or inspect context cache state.
Inputs: action, auth_token, content_id. Returns: object
infra_configureInfra ConfigureAIdempotent
Reconfigure the server at runtime — preset, model, thinking level, or temperature.
Changes take effect immediately for all subsequent tool calls.
Inputs: model, preset, auth_token, temperature, +1 more. Returns: object
job_cancelJob CancelA
Cancel queued video work or request its owner stop at the next checkpoint.
Inputs: job_id (required). Returns: object
job_statusJob StatusBRead-only
Read retained job identity, outcomes and current byte attestation.
Inputs: job_id (required). Returns: object
knowledge_askKnowledge AskARead-onlyIdempotent
Ask a question and get an AI-generated answer grounded in stored knowledge.
Uses Weaviate AsyncQueryAgent in ask mode to synthesize an answer from objects across knowledge collections, with source citations.
Inputs: query (required), collections. Returns: object
knowledge_fetchKnowledge FetchARead-onlyIdempotent
Fetch a single object by UUID from a knowledge collection.
Inputs: object_id (required), collection (required). Returns: object
knowledge_ingestKnowledge IngestA
Manually insert data into a knowledge collection.
Properties are validated against the collection schema — unknown keys are rejected with allowed name:type pairs.
Tip: call knowledge_schema(collection=...) first to see expected properties.
Inputs: collection (required), properties (required). Returns: object
knowledge_queryKnowledge QueryARead-onlyIdempotent
[DEPRECATED] Search knowledge store using natural language.
Deprecated: Use knowledge_search instead, which now includes Cohere reranking and Flash summarization for better results with lower token usage.
Uses Weaviate AsyncQueryAgent in search mode for intelligent object retrieval.
Inputs: query (required), limit, collections. Returns: object
knowledge_schemaKnowledge SchemaARead-onlyIdempotent
Return property schemas for knowledge collections.
Reads from local CollectionDef objects — no Weaviate connection needed. Use this before knowledge_ingest to discover expected property names and types.
Inputs: collection. Returns: object
knowledge_searchKnowledge SearchARead-onlyIdempotent
Search across knowledge collections using hybrid, semantic, or keyword mode.
Searches specified or all collections. Results are merged and sorted by score. Filters are collection-aware: conditions are skipped for collections that lack the relevant property.
Inputs: query (required), alpha, limit, date_to, +7 more. Returns: object
knowledge_statsKnowledge StatsBRead-onlyIdempotent
Get object counts per collection, optionally grouped by a property.
Inputs: group_by, collection. Returns: object
media_acquireMedia AcquireC
Acquire bounded media into private owned storage with exact byte identity.
Inputs: source (required). Returns: object
media_analyze_musicMedia Analyze MusicA
Infer timed musical sections and properties from actual decoded audio.
Instruments, mood, tags, tempo, key and meter remain model interpretations. Submitted PCM clocks and hashes identify the evidence for each section.
Inputs: request (required). Returns: object
media_asset_removeMedia Asset RemoveADestructive
Remove one verified owned asset and invalidate its source-dependent caches.
Inputs: asset_id (required). Returns: object
media_assets_listMedia Assets ListBRead-onlyIdempotent
List a bounded page with fresh byte checks and explicit unhealthy states.
Inputs: limit, offset. Returns: object
media_caption_eventsMedia Caption EventsC
Caption timed audio, visual and combined occurrences with submitted support.
Inputs: request (required). Returns: object
media_count_eventsMedia Count EventsB
Count retained inferred occurrences of a target in bounded source windows.
Counts concern admitted records in submitted evidence. Overlapping records remain distinct, and sampled frames leave gaps where events may be missed.
Inputs: request (required). Returns: object
media_edit_footageMedia Edit FootageA
Edit bounded local footage through a measured prepare and approval workflow.
Prepare retains source intervals, actual frame clocks, grade/audio decisions and scene previews. Assemble requires the exact prepared manifest and every scene artifact approval, then gates full decoding, timeline, black spans and required audio loudness before delivery. Declared beat grids and approvals remain caller assertions; technical checks do not establish visual semantics. FFmpeg and Pillow must already be installed. No provider is invoked.
Inputs: request (required), include_image. Returns: object
media_ground_eventsMedia Ground EventsC
Ground a query in source-supported intervals with uncalibrated scores.
Inputs: request (required). Returns: object
media_infoMedia InfoA
Inspect exact local media bytes, stream metadata, rotation, audio and chapters.
Uses independently installed FFprobe and a bounded private stable snapshot. This performs no provider upload and does not establish watched coverage.
Inputs: file_path (required). Returns: object
media_metadataMedia MetadataBRead-onlyIdempotent
Retrieve explicit metadata without downloading video, inference or upload.
Inputs: source (required). Returns: object
media_perceiveMedia PerceiveA
Infer separate spoken and visible source evidence across bounded AV windows.
Local dry plans prepare exact sampled frames and decoded PCM with no provider calls. Submission requires the exact workflow grant and configured account. Model intervals map back to source seconds; sampled images leave visual gaps.
Inputs: request (required). Returns: object
media_source_inspectMedia Source InspectARead-onlyIdempotent
Inspect source syntax and capabilities without contacting or downloading it.
Inputs: source (required). Returns: object
mhs_discoverMhs DiscoverBRead-onlyIdempotent
Discover registered simulated devices and their explicit capabilities.
Inputs: tag, device_type. Returns: object
mhs_health_checkMhs Health CheckCIdempotent
Inspect fresh simulated health and acknowledge a failed-write health gate.
Inputs: device_id. Returns: object
mhs_meta_infoMhs Meta InfoBRead-onlyIdempotent
Inspect current device values, directions and effective simulated limits.
Inputs: device_id (required). Returns: object
mhs_readMhs ReadC
Read declared simulated sensors or retain independently generated image bytes.
Inputs: params (required), device_id (required), capability (required). Returns: object
mhs_resetMhs ResetBIdempotent
Latch a simulated emergency stop or perform separately authorized recovery.
Inputs: mode (required), device_id (required), command_id (required). Returns: object
mhs_writeMhs WriteAIdempotent
Apply one atomic simulated write or return its exact authority proposal.
Soft overrides and confirmation-required operations need separate host approval bound to this command, state and limits. confirm=True grants no approval. Terminal IDs replay their receipt. An explicitly resumed pending proposal may attempt its first action after exact host approval.
Inputs: params (required), device_id (required), capability (required), command_id (required), +1 more. Returns: object
provider_capabilitiesProvider CapabilitiesARead-only
Inspect configured operation support and reject unsupported selections.
Profiles declare support; inspection performs no connectivity or model test.
Inputs: backend, required. Returns: object
research_assess_evidenceResearch Assess EvidenceBRead-only
Assess a claim against sources, returning evidence tier and confidence.
Inputs: claim (required), sources (required), context. Returns: object
research_author_searchResearch Author SearchBRead-only
Search for academic authors on Semantic Scholar.
Returns author profiles with affiliation, paper count, citation count, and h-index.
Inputs: query (required), limit. Returns: object
research_deepResearch DeepARead-only
Run three-phase model-only synthesis without external source retrieval.
Phases: Scope Definition -> Evidence Collection -> Synthesis. Model tiers remain proposals. CONFIRMED and STRONG INDICATOR proposals become UNKNOWN because this route neither retrieves nor verifies sources.
Inputs: topic (required), scope, thinking_level. Returns: object
research_documentResearch DocumentARead-only
Run multi-phase deep research grounded in source documents.
Phases: Document Mapping -> Evidence Extraction -> Cross-Reference -> Synthesis. Every claim is labeled with evidence tiers and cited back to source documents.
Inputs: instruction (required), urls, scope, file_paths, +1 more. Returns: object
research_executeResearch ExecuteA
Prepare or execute source-preserving research under one global allowance.
Dry runs make no provider or URL calls. Supplied originals are byte checked. URL execution requires source-access authorization; inference requires submission authorization. Google hosted research remains a separate route. Unknown price ceilings fail closed. Model tiers never establish CONFIRMED.
Inputs: request (required). Returns: object
research_paper_citationsResearch Paper CitationsCRead-only
Get citations or references for a paper.
Inputs: paper_id (required), limit, direction. Returns: object
research_paper_detailsResearch Paper DetailsARead-only
Get detailed metadata for a specific paper.
Accepts Semantic Scholar IDs, DOIs (prefix with DOI:), or ArXiv IDs (prefix with ArXiv:).
Inputs: paper_id (required). Returns: object
research_paper_recommendationsResearch Paper RecommendationsBRead-only
Get paper recommendations based on seed papers.
Uses Semantic Scholar's recommendation engine to find related papers.
Inputs: seed_paper_ids (required), limit. Returns: object
research_paper_searchResearch Paper SearchARead-only
Search academic papers on Semantic Scholar.
Returns papers with metadata, abstracts, citation counts, and TL;DR summaries. Supports filtering by field of study, year range, and open access availability.
Inputs: query (required), limit, year_range, fields_of_study, +1 more. Returns: object
research_planResearch PlanARead-only
Generate a multi-agent research orchestration plan.
Returns a phased blueprint with task decomposition and model assignments. Does NOT spawn agents — provides the blueprint for the caller.
Inputs: topic (required), scope, available_agents. Returns: object
research_webResearch WebA
Launch a Gemini Deep Research Agent for autonomous web-grounded research.
The agent plans its own research, searches the web (~80-160 queries), reads sources, and produces a cited markdown report. Runs in background; poll with research_web_status. Provider usage is billed; runtime varies.
Inputs: topic (required), job_id, output_format. Returns: object
research_web_cancelResearch Web CancelA
Cancel a running Deep Research task.
Sends a cancel request to the Interactions API and cleans up local tracking state. Useful for stopping a task before further provider usage.
Inputs: interaction_id (required). Returns: object
research_web_followupResearch Web FollowupA
Ask a follow-up question about a completed Deep Research report.
Uses previous_interaction_id to maintain context from the original research. Synchronous — follow-ups are fast (no background needed).
Inputs: question (required), interaction_id (required), job_id. Returns: object
research_web_statusResearch Web StatusBRead-only
Poll or retrieve a Deep Research task.
Returns the full report with sources when completed, or current status if still in progress. Auto-stores completed reports to Weaviate.
Inputs: interaction_id (required). Returns: object
reverse_search_frameReverse Search FrameA
Plan or reverse-search one captured frame using Serper Lens.
Execution publicly uploads the PNG to Uguu, verifies hosted bytes, then submits its URL to Lens. It requires both explicit grants and an enabled Serper credential. Similarity suggestions never assert entity identity; compare appearance, text and source context before reporting an identity. Publication retention/deletion and live service schemas remain unqualified.
Inputs: request (required). Returns: object
session_memorySession MemoryADestructive
Inspect original history or read/edit/delete one explicitly scoped derived profile.
History remains recoverable after active-session expiry when SQLite persistence is configured. Large inline-media pages export exact bounded private JSON. Memory edits require the current revision; deletion clears only derived text. Caller-selected scope is isolation, not an authenticated multi-user identity. No provider, inference, upload or global-memory search occurs.
Inputs: request (required). Returns: object
source_ingestSource IngestAIdempotent
Retain original bytes and extract bounded elements with explicit source positions.
PDF requires separately installed Poppler; DOCX, Markdown, HTML, plain text and PCM16 WAV use local parsers. Unsupported OCR, PDF table reconstruction, image pixels and speech are reported explicitly. Empty/error extraction fails. Identical source/revision/parser work reuses its durable job. Extraction does not index, call a model, upload to a parser service or prove factual claims.
Inputs: request (required). Returns: object
source_ingest_cancelSource Ingest CancelBIdempotent
Request cancellation without promoting an unfinished extraction to success.
The owner acknowledges cancellation after its bounded parser has joined; this request alone does not prove that a running native process has stopped.
Inputs: job_id (required). Returns: object
source_ingest_readSource Ingest ReadCRead-onlyIdempotent
Recheck a retained ingestion job and read exact located extraction bytes.
Inputs: job_id (required). Returns: object
text_generateText GenerateA
Generate structured text using one selected compatible HTTP endpoint.
No default Gemini/vision account is borrowed. Sources are untrusted text; this operation cannot accept files, media, tools or grounded search. Missing input-token/currency meters cause requested hard ceilings to fail.
Inputs: request (required). Returns: object
twelvelabs_callTwelvelabs CallADestructive
Plan or perform one bounded TwelveLabs asset/index/search/embed/entity/analysis operation.
Disabled by default. Dry plans read only selected fenced local media and make no DNS, provider or database calls. Execution requires explicit BYOK/network authority; media disclosure and deletion/cancellation have separate grants. Each call makes at most one HTTP request with no retries or automatic polling. Reusing an identical job_id reads the attested local receipt without resending. Remote readiness, exact clip seconds and IDs stay provider observations.
Inputs: request (required). Returns: object
video_analyzeVideo AnalyzeAIdempotent
Analyze a video (YouTube URL or local file) with any instruction.
Provide exactly one of url or file_path. Uses Gemini's structured output for reliable JSON responses. Pass a custom output_schema to control the response shape, or use the default VideoResult schema.
When strict_contract=True, runs the full contract pipeline: analysis with strict Pydantic models, parallel strategy/concept-map generation, artifact rendering, and structural quality gates. Factual/media review and observed coverage remain explicit pending/unknown fields. Returns richer output but takes longer.
Inputs: fps, url, dry_run, file_path, +11 more. Returns: object
video_analyze_windowsVideo Analyze WindowsC
Plan or execute contiguous source windows with retained partial outcomes.
Inputs: request (required), execution_budget (required), job_id, dry_run, +1 more. Returns: object
video_batch_analyzeVideo Batch AnalyzeARead-onlyIdempotent
Analyze all video files in a directory concurrently.
Scans the directory for supported video files (mp4, webm, mov, avi, mkv, mpeg, wmv, 3gpp), then analyzes each with the given instruction using bounded concurrency (3 parallel Gemini calls). Discovery rejects more than 5,000 entry visits; narrow the directory or pattern to stay within that bound.
Inputs: directory (required), job_id, max_files, instruction, +4 more. Returns: object
video_clip_exportVideo Clip ExportA
Export a bounded source clip and verify its encoded output and manifest.
Requested bounds, selected original PTS and actual encoded timing remain separate. This operation preserves the original and invokes no provider.
Inputs: request (required). Returns: object
video_clip_selectVideo Clip SelectC
Export a whole or one-sided bounded video selection using measured clip timing.
Inputs: request (required). Returns: object
video_commentsVideo CommentsARead-onlyIdempotent
Fetch top YouTube comments sorted by relevance.
Returns comment text, like count, and author for each comment. Costs 1+ YouTube API units, 0 Gemini units.
Inputs: url (required), max_comments. Returns: object
video_continue_sessionVideo Continue SessionB
Continue analysis within an existing video session.
Inputs: prompt (required), session_id (required), scope. Returns: object
video_create_sessionVideo Create SessionA
Create a persistent session for multi-turn video exploration.
Provide exactly one of url or file_path. When download=True and the
source is YouTube, the video is downloaded via yt-dlp, uploaded to the
Gemini File API, and context-cached for fast multi-turn use.
Inputs: url, scope, download, file_path, +1 more. Returns: object
video_deduplicate_framesVideo Deduplicate FramesC
Compare bounded source frames without dropping similar or failed candidates.
Inputs: request (required), include_image. Returns: object
video_detect_scenesVideo Detect ScenesC
Detect hard visual cuts and return contiguous intervals with actual source PTS.
Inputs: request (required). Returns: object
video_frameVideo FrameB
Retrieve a source frame with requested time, original decoded PTS and actual delta.
Times are relative to the measured container presentation origin. Precise selection returns the first decoded frame at or after the request. Keyframe selection reports its measured approximation and selection method.
Inputs: file_path (required), time_seconds (required), crop_box, selection, +2 more. Returns: object
video_frame_by_queryVideo Frame By QueryB
Select a frame by lexical transcript match, requiring the exact video digest.
Segment times use the container-relative presentation clock. Ties choose the earliest segment. A zero-token match abstains. The supplied transcript remains unverified; the extracted frame carries measured PTS.
Inputs: query (required), file_path (required), transcript (required), max_pixels, +1 more. Returns: object
video_framesVideo FramesC
Inspect a bounded burst/window or timestamped contact sheet without a watched claim.
Dense filmstrips require a narrow window and explicit FPS. Selection retains changed text; it performs no similarity pruning or OCR. Sheet tiles carry actual source times/digests. Scene/keyframe sampling may miss short events.
Inputs: file_path (required), fps, mode, crop_box, +5 more. Returns: object
video_metadataVideo MetadataARead-onlyIdempotent
Fetch YouTube video metadata without Gemini analysis.
Returns title, description, view/like/comment counts, duration, tags, channel info, category, and language. Costs 1 YouTube API unit, 0 Gemini units.
Inputs: url (required). Returns: object
video_note_createVideo Note CreateADestructive
Create a source-linked illustrated tutorial PDF or validate a read-only dry plan.
Actual generation retains exact source PNGs, page rasters and a manifest/receipt. Missing illustrations or AV inference preserve supplied text with warnings. Deterministic PDF checks do not establish factual or human review.
Inputs: request (required). Returns: object
video_playlistVideo PlaylistARead-onlyIdempotent
Get video IDs and titles from a YouTube playlist.
Results can be passed to video_analyze for batch analysis. Costs 1 YouTube API unit per page (max 50 items/page).
Inputs: url (required), max_items. Returns: object
video_storyboardVideo StoryboardC
Create a measured source storyboard with literal absolute source-clock labels.
Inputs: request (required), include_image. Returns: object
vision_chatVision ChatA
Analyze, identify or compare exact images and bounded source video samples.
Dry plans perform local preparation with zero provider calls. A live request requires explicit workflow authorization and a configured model/account. Interpretation remains model inference, separately from source byte lineage.
Inputs: request (required). Returns: object
vision_groundingVision GroundingB
Locate objects with strict inferred boxes and optional actual original-pixel crops.
Normalized1000 boxes map through the exact transmitted preparation to original stored/oriented pixels. Crop byte/geometry verification does not verify that the model identified the object correctly. Empty detections remain valid abstentions.
Inputs: request (required). Returns: object
vision_ocrVision OcrB
Infer visible text from exact prepared images or precise source frames.
Model transcription is labeled inference. The separate image_ocr operation selects local Tesseract/Vision observations. No automatic fallback occurs.
Inputs: request (required). Returns: object
web_extractWeb ExtractA
Acquire one checked UTF-8 page or a labeled provider-returned representation.
Dry plans make no DNS/HTTP requests. Direct redirects stay within the source allowlist. Provider services' remote page peers, redirects and original-byte identity remain unverified; returned text cannot authorize further actions.
Inputs: request (required). Returns: object
web_searchWeb SearchCRead-only
Search the web using Gemini's built-in Google Search grounding.
Inputs: query (required), num_results. Returns: object
web_search_providerWeb Search ProviderA
Plan or perform one explicitly selected optional search request.
Automatic selection uses configured credential-ready Serper, Tavily, Exa, then Serply. Missing pinned keys and failed requests never cause fallback. Provider snippets remain untrusted data and do not establish factual support.
Inputs: request (required). Returns: object
youtube_channel_catalogYoutube Channel CatalogARead-onlyIdempotent
List one channel uploads page and retain any continuation token.
A channel video count and playlist reported total are provider metadata. The returned items establish this page's coverage; no video bytes are fetched.
Inputs: reference (required), max_items, page_token. Returns: object
youtube_channel_inspectYoutube Channel InspectBRead-onlyIdempotent
Inspect channel metadata through YouTube Data API without downloading media.
Inputs: reference (required). Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 90 tools
With 90 tools across knowledge, research, video, audio, image, web, and device simulation, many boundaries overlap: video_analyze/content_analyze/media_perceive/vision_chat, research_web/research_deep/research_document/research_execute, and video_frame/video_frames/video_frame_by_query are especially confusable. The detailed descriptions help, but an agent still faces high misselection risk because many distinctions are workflow-specific and subtle.
All names use snake_case and largely follow a domain-prefixed operation pattern (knowledge_ingest, video_analyze, research_web_status), which is readable and predictable by area. However, there are mixed noun-only and verb styles (knowledge_stats, media_info, job_status) and brand/acronym prefixes (mhs_, twelvelabs_), so it is not a perfectly uniform verb_noun convention.
90 tools is far beyond the typical 3-15 scoped set and exceeds even the 25+ heavy threshold. The server spans many unrelated domains, making the surface unwieldy and creating extreme cognitive load for tool selection.
The surface covers research, knowledge, video/image/audio processing, web, and media assets with lifecycle operations such as ingest, search, acquire, export, edit, and cancel. Minor gaps exist, including knowledge object update/delete and collection management, but most workflows have supporting tools.
Maintenance
Related MCP Connectors
Unified 15-tool SEO, SERP, clean Markdown & GEO (LLM citation Share-of-Voice) MCP server.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
MCP server for Clipkit — gives AI agents a video toolbox via the Clipkit schema.
MCP server for Google Veo AI video generation
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA local MCP server for extracting YouTube video transcripts, metadata, and performing visual analysis using Gemini Vision or local Whisper models. It enables users to process video content through various tools for subtitle retrieval and frame analysis.23 npmMIT
- AlicenseNot gradedqualityDmaintenanceMCP server enabling interaction with Twitter, YouTube, Instagram, and video processing via 31 tools, with local Whisper transcription and frame extraction for visual verification.8MIT
- AlicenseAqualityBmaintenanceMCP server for AI-powered research using Gemini. Provides fast grounded web search, deep autonomous research, URL extraction, and session management.9154 PyPI10MIT
- AlicenseAqualityBmaintenanceMCP server for extracting structured intelligence from YouTube channels and videos — transcripts, topics, and competitive signals for AI-powered research workflows.168 npmMIT