Analyze a video (YouTube URL or local file) with any instruction.

Provide exactly one of url or file_path. Uses Gemini's structured output for reliable JSON responses. Pass a custom output_schema to control the response shape, or use the default VideoResult schema.

When strict_contract=True, runs the full contract pipeline: analysis with strict Pydantic models, parallel strategy/concept-map generation, artifact rendering, and structural quality gates. Factual/media review and observed coverage remain explicit pending/unknown fields. Returns richer output but takes longer.

Create a persistent session for multi-turn video exploration.

Provide exactly one of url or file_path. When download=True and the source is YouTube, the video is downloaded via yt-dlp, uploaded to the Gemini File API, and context-cached for fast multi-turn use.

Continue analysis within an existing video session.

Analyze all video files in a directory concurrently.

Scans the directory for supported video files (mp4, webm, mov, avi, mkv, mpeg, wmv, 3gpp), then analyzes each with the given instruction using bounded concurrency (3 parallel Gemini calls). Discovery rejects more than 5,000 entry visits; narrow the directory or pattern to stay within that bound.

Plan or execute contiguous source windows with retained partial outcomes.

Run three-phase model-only synthesis without external source retrieval.

Phases: Scope Definition -> Evidence Collection -> Synthesis. Model tiers remain proposals. CONFIRMED and STRONG INDICATOR proposals become UNKNOWN because this route neither retrieves nor verifies sources.

Generate a multi-agent research orchestration plan.

Returns a phased blueprint with task decomposition and model assignments. Does NOT spawn agents — provides the blueprint for the caller.

Assess a claim against sources, returning evidence tier and confidence.

Run multi-phase deep research grounded in source documents.

Phases: Document Mapping -> Evidence Extraction -> Cross-Reference -> Synthesis. Every claim is labeled with evidence tiers and cited back to source documents.

Launch a Gemini Deep Research Agent for autonomous web-grounded research.

The agent plans its own research, searches the web (~80-160 queries), reads sources, and produces a cited markdown report. Runs in background; poll with research_web_status. Provider usage is billed; runtime varies.

Poll or retrieve a Deep Research task.

Returns the full report with sources when completed, or current status if still in progress. Auto-stores completed reports to Weaviate.

Ask a follow-up question about a completed Deep Research report.

Uses previous_interaction_id to maintain context from the original research. Synchronous — follow-ups are fast (no background needed).

Cancel a running Deep Research task.

Sends a cancel request to the Interactions API and cleans up local tracking state. Useful for stopping a task before further provider usage.

Search academic papers on Semantic Scholar.

Returns papers with metadata, abstracts, citation counts, and TL;DR summaries. Supports filtering by field of study, year range, and open access availability.

Get detailed metadata for a specific paper.

Accepts Semantic Scholar IDs, DOIs (prefix with DOI:), or ArXiv IDs (prefix with ArXiv:).

Get citations or references for a paper.

Get paper recommendations based on seed papers.

Uses Semantic Scholar's recommendation engine to find related papers.

Search for academic authors on Semantic Scholar.

Returns author profiles with affiliation, paper count, citation count, and h-index.

Prepare or execute source-preserving research under one global allowance.

Dry runs make no provider or URL calls. Supplied originals are byte checked. URL execution requires source-access authorization; inference requires submission authorization. Google hosted research remains a separate route. Unknown price ceilings fail closed. Model tiers never establish CONFIRMED.

Analyze content (file, URL, or text) with any instruction.

Provide exactly one of file_path, url, or text. Uses Gemini's structured output for reliable JSON responses. Pass a custom output_schema to control the response shape, or use the default ContentResult schema.

Extract structured data from content using a JSON Schema.

Uses Gemini's response_json_schema for guaranteed structured output.

Analyze multiple content files from a directory or explicit file list.

Rejects discovery over 5,000 entry visits and compare payloads over DOC_MAX_DOWNLOAD_BYTES. Supports two modes: 'compare' sends files in a single call for cross-document analysis, 'individual' analyzes each file separately with bounded concurrency (3 parallel calls).

Search the web using Gemini's built-in Google Search grounding.

Manage the analysis cache — stats, list, clear, or inspect context cache state.

Reconfigure the server at runtime — preset, model, thinking level, or temperature.

Changes take effect immediately for all subsequent tool calls.

Fetch YouTube video metadata without Gemini analysis.

Returns title, description, view/like/comment counts, duration, tags, channel info, category, and language. Costs 1 YouTube API unit, 0 Gemini units.

Fetch top YouTube comments sorted by relevance.

Returns comment text, like count, and author for each comment. Costs 1+ YouTube API units, 0 Gemini units.

Get video IDs and titles from a YouTube playlist.

Results can be passed to video_analyze for batch analysis. Costs 1 YouTube API unit per page (max 50 items/page).

Inspect channel metadata through YouTube Data API without downloading media.

List one channel uploads page and retain any continuation token.

A channel video count and playlist reported total are provider metadata. The returned items establish this page's coverage; no video bytes are fetched.

Search across knowledge collections using hybrid, semantic, or keyword mode.

Searches specified or all collections. Results are merged and sorted by score. Filters are collection-aware: conditions are skipped for collections that lack the relevant property.

Find semantically related objects using near-object vector search.

Get object counts per collection, optionally grouped by a property.

Fetch a single object by UUID from a knowledge collection.

Manually insert data into a knowledge collection.

Properties are validated against the collection schema — unknown keys are rejected with allowed name:type pairs.

Tip: call knowledge_schema(collection=...) first to see expected properties.

Ask a question and get an AI-generated answer grounded in stored knowledge.

Uses Weaviate AsyncQueryAgent in ask mode to synthesize an answer from objects across knowledge collections, with source citations.

[DEPRECATED] Search knowledge store using natural language.

Deprecated: Use knowledge_search instead, which now includes Cohere reranking and Flash summarization for better results with lower token usage.

Uses Weaviate AsyncQueryAgent in search mode for intelligent object retrieval.

Return property schemas for knowledge collections.

Reads from local CollectionDef objects — no Weaviate connection needed. Use this before knowledge_ingest to discover expected property names and types.

Crop local PNG pixels without model inference or provider uploads.

Requires an independently installed FFmpeg and respects LOCAL_FILE_ACCESS_ROOT. A source hash binds the crop to bytes; it does not establish factual support or whether the input was synthetic. Large outputs retain metadata/path only.

Inspect exact local media bytes, stream metadata, rotation, audio and chapters.

Uses independently installed FFprobe and a bounded private stable snapshot. This performs no provider upload and does not establish watched coverage.

Return a bounded local still image as inspectable PNG and exact source metadata.

Animated inputs expose one still with an explicit limitation. Native output may be processed by the host's model; this server invokes no model or upload.

Retrieve a source frame with requested time, original decoded PTS and actual delta.

Times are relative to the measured container presentation origin. Precise selection returns the first decoded frame at or after the request. Keyframe selection reports its measured approximation and selection method.

Inspect a bounded burst/window or timestamped contact sheet without a watched claim.

Dense filmstrips require a narrow window and explicit FPS. Selection retains changed text; it performs no similarity pruning or OCR. Sheet tiles carry actual source times/digests. Scene/keyframe sampling may miss short events.

Select a frame by lexical transcript match, requiring the exact video digest.

Segment times use the container-relative presentation clock. Ties choose the earliest segment. A zero-token match abstains. The supplied transcript remains unverified; the extracted frame carries measured PTS.

Read retained job identity, outcomes and current byte attestation.

Cancel queued video work or request its owner stop at the next checkpoint.

Inspect source syntax and capabilities without contacting or downloading it.

Retrieve explicit metadata without downloading video, inference or upload.

Acquire bounded media into private owned storage with exact byte identity.

List a bounded page with fresh byte checks and explicit unhealthy states.

Remove one verified owned asset and invalidate its source-dependent caches.

Edit a source image or decoded video frame and persist its exact manifest.

Coordinates use the oriented source grid before crop and resize. Requires the optional images extra; frame selection also requires local FFmpeg. Source-derived edits and annotations retain explicit provenance.

Read an exact manifest after restart and verify its source and artifact bytes.

Observe text and geometry using an explicitly selected local OCR engine.

Source OCR and detected geometry do not establish semantic table structure or factual support. Missing local runtimes return an explicit limitation.

Export a bounded source clip and verify its encoded output and manifest.

Requested bounds, selected original PTS and actual encoded timing remain separate. This operation preserves the original and invokes no provider.

Analyze, identify or compare exact images and bounded source video samples.

Dry plans perform local preparation with zero provider calls. A live request requires explicit workflow authorization and a configured model/account. Interpretation remains model inference, separately from source byte lineage.

Infer visible text from exact prepared images or precise source frames.

Model transcription is labeled inference. The separate image_ocr operation selects local Tesseract/Vision observations. No automatic fallback occurs.

Locate objects with strict inferred boxes and optional actual original-pixel crops.

Normalized1000 boxes map through the exact transmitted preparation to original stored/oriented pixels. Crop byte/geometry verification does not verify that the model identified the object correctly. Empty detections remain valid abstentions.

Plan or submit one image and preserve actual mask and overlay artifacts.

Scores and masks are unverified proposals. Service selection and checkpoint declarations do not attest a loaded model or establish redistribution rights.

Edit bounded local footage through a measured prepare and approval workflow.

Prepare retains source intervals, actual frame clocks, grade/audio decisions and scene previews. Assemble requires the exact prepared manifest and every scene artifact approval, then gates full decoding, timeline, black spans and required audio loudness before delivery. Declared beat grids and approvals remain caller assertions; technical checks do not establish visual semantics. FFmpeg and Pillow must already be installed. No provider is invoked.

Infer separate spoken and visible source evidence across bounded AV windows.

Local dry plans prepare exact sampled frames and decoded PCM with no provider calls. Submission requires the exact workflow grant and configured account. Model intervals map back to source seconds; sampled images leave visual gaps.

Caption timed audio, visual and combined occurrences with submitted support.

Count retained inferred occurrences of a target in bounded source windows.

Counts concern admitted records in submitted evidence. Overlapping records remain distinct, and sampled frames leave gaps where events may be missed.

Ground a query in source-supported intervals with uncalibrated scores.

Infer timed musical sections and properties from actual decoded audio.

Instruments, mood, tags, tempo, key and meter remain model interpretations. Submitted PCM clocks and hashes identify the evidence for each section.

Detect hard visual cuts and return contiguous intervals with actual source PTS.

Create a measured source storyboard with literal absolute source-clock labels.

Compare bounded source frames without dropping similar or failed candidates.

Compare local audio candidates while retaining silence, errors and uncertainty.

Requires the optional audio extra and local FFmpeg. Similarity does not establish equal speech, speaker identity or permission to delete evidence.

Export a whole or one-sided audio selection to a unique private 16kHz mono WAV.

Originals and prior exports remain unchanged. Caller-selected output paths and overwriting are rejected. Use image_manifest_read for verified restart.

Export a whole or one-sided bounded video selection using measured clip timing.

Generate structured text using one selected compatible HTTP endpoint.

No default Gemini/vision account is borrowed. Sources are untrusted text; this operation cannot accept files, media, tools or grounded search. Missing input-token/currency meters cause requested hard ceilings to fail.

Inspect configured operation support and reject unsupported selections.

Profiles declare support; inspection performs no connectivity or model test.

Plan or perform one explicitly selected optional search request.

Automatic selection uses configured credential-ready Serper, Tavily, Exa, then Serply. Missing pinned keys and failed requests never cause fallback. Provider snippets remain untrusted data and do not establish factual support.

Acquire one checked UTF-8 page or a labeled provider-returned representation.

Dry plans make no DNS/HTTP requests. Direct redirects stay within the source allowlist. Provider services' remote page peers, redirects and original-byte identity remain unverified; returned text cannot authorize further actions.

Plan or reverse-search one captured frame using Serper Lens.

Execution publicly uploads the PNG to Uguu, verifies hosted bytes, then submits its URL to Lens. It requires both explicit grants and an enabled Serper credential. Similarity suggestions never assert entity identity; compare appearance, text and source context before reporting an identity. Publication retention/deletion and live service schemas remain unqualified.

Plan or perform one bounded TwelveLabs asset/index/search/embed/entity/analysis operation.

Disabled by default. Dry plans read only selected fenced local media and make no DNS, provider or database calls. Execution requires explicit BYOK/network authority; media disclosure and deletion/cancellation have separate grants. Each call makes at most one HTTP request with no retries or automatic polling. Reusing an identical job_id reads the attested local receipt without resending. Remote readiness, exact clip seconds and IDs stay provider observations.

Measure audio or invoke one separately installed operator-pinned native DSP workflow.

Local analysis preserves stereo, finite silence/abstentions and absolute clipping intervals. A/B uses reference-minus-primary differences with units. Optional Juzzy modes preserve harmonic/rhythm/masking/section observations; Ferrous receives an explicit mono derivation and retains heuristic quality, content and fingerprint output. Standards and semantic claims are unverified. Reusing an exact job_id reads its attested artifacts without launching again. Use job_status for durable status, including interrupted/failed work.

Extract exact caption intervals or infer speech through an explicit backend.

Supplied cue and word times remain source assertions. Model-derived words and speaker labels remain inference. Caption selection precedes ASR; local-only selection forbids cloud requests. Readback requires the retained receipt digest and exact source identity. No local model is installed.

Inspect original history or read/edit/delete one explicitly scoped derived profile.

History remains recoverable after active-session expiry when SQLite persistence is configured. Large inline-media pages export exact bounded private JSON. Memory edits require the current revision; deletion clears only derived text. Caller-selected scope is isolation, not an authenticated multi-user identity. No provider, inference, upload or global-memory search occurs.

Retain original bytes and extract bounded elements with explicit source positions.

PDF requires separately installed Poppler; DOCX, Markdown, HTML, plain text and PCM16 WAV use local parsers. Unsupported OCR, PDF table reconstruction, image pixels and speech are reported explicitly. Empty/error extraction fails. Identical source/revision/parser work reuses its durable job. Extraction does not index, call a model, upload to a parser service or prove factual claims.

Recheck a retained ingestion job and read exact located extraction bytes.

Request cancellation without promoting an unfinished extraction to success.

The owner acknowledges cancellation after its bounded parser has joined; this request alone does not prove that a running native process has stopped.

Discover registered simulated devices and their explicit capabilities.

Inspect fresh simulated health and acknowledge a failed-write health gate.

Inspect current device values, directions and effective simulated limits.

Read declared simulated sensors or retain independently generated image bytes.

Apply one atomic simulated write or return its exact authority proposal.

Soft overrides and confirmation-required operations need separate host approval bound to this command, state and limits. confirm=True grants no approval. Terminal IDs replay their receipt. An explicitly resumed pending proposal may attempt its first action after exact host approval.

Latch a simulated emergency stop or perform separately authorized recovery.

Create a source-linked illustrated tutorial PDF or validate a read-only dry plan.

Actual generation retains exact source PNGs, page rasters and a manifest/receipt. Missing illustrations or AV inference preserve supplied text with warnings. Deterministic PDF checks do not establish factual or human review.