Skip to main content
Glama

Audio To Video

audio_to_video

Generates video conditioned on an audio file and text prompt, producing synchronized audio-visual output with optional first and last frame anchor images.

Instructions

Generate video from audio and text prompt.

Creates videos conditioned on audio content. The audio drives the video generation, allowing synchronized audio-visual output. Optionally accepts anchor frame images.

IMPORTANT: Check model specifications using get_available_models() before calling. Pay attention to features.supports_guidance, features.supports_steps, and model limits.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "prompt": "<prompt>",
      "model": "<model>",
      "audio": "<audio>"
    },
    "name": "audio_to_video"
  }
}

Input Schema

NameRequiredDescriptionDefault
fpsNoFrames per second. Defaults to the model's own value (see the model list); most models fix it
seedNoRandom seed (-1 for random)
audioYesAudio file as URL, data URI (data:audio/mpeg;base64,...), or base64 string. Conditions the video generation on the audio content
modelYesVideo generation model name (must support audio2video inference type)
stepsNoInference steps. Only used if model's features.supports_steps is true. Limits per model (see the model list)
widthNoVideo width in pixels. Defaults to the model's own value (see the model list)
framesNoNumber of video frames. Defaults to the model's own value (see the model list); each model has its own range
heightNoVideo height in pixels. Defaults to the model's own value (see the model list)
promptYesText prompt for video generation
guidanceNoGuidance scale. Only used if model's features.supports_guidance is true. Limits per model (see the model list)
webhook_urlNoHTTPS URL for deAPI to call when this job finishes. Notifies your own systems; the tool still returns the result here
webhook_secretNo32-255 char key deAPI signs the callback with (HMAC-SHA256). Needs webhook_url. A secret given in chat stays in the transcript
negative_promptNoThings to exclude (optional)
last_frame_imageNoLast frame anchor image (URL, data URI, or base64, optional)
first_frame_imageNoFirst frame anchor image (URL, data URI, or base64, optional)

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.6.3

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full disclosure burden and does add real behavior: the audio conditions and drives generation for synchronized output, anchor frames are optional, and several parameters are gated by per-model feature flags. It still omits operational traits such as whether the job is asynchronous, how long generation takes, and that results are also retrievable via check_job_status, so it is not exhaustive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and the important model-check warning are front-loaded, and the four short paragraphs are easy to scan. The first two sentences overlap ('Generate video from audio...' / 'Creates videos conditioned on audio content'), which is mild redundancy rather than filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 15-parameter, 3-required generation tool with an output schema and no annotations, the description covers purpose, modality, anchor frames, and the model-compatibility prerequisite. Gaps remain around job lifecycle and cost (a sibling price tool exists), but the essentials for correct invocation are present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; the description exceeds it by telling the agent where to resolve the model-dependent parameters (guidance, steps, and model limits) via get_available_models, which the schema only states as conditions without saying how to verify them. It adds little about the other eleven parameters, but that is acceptable given the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('Generate video from audio and text prompt') and states the conditioning modality ('videos conditioned on audio content'), which is what distinguishes this tool from the text_to_video and image_to_video siblings without naming them explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a concrete prerequisite workflow: 'Check model specifications using get_available_models() before calling,' and points at the exact feature flags (supports_guidance, supports_steps) and limits to inspect. It never states when to prefer this over text_to_video or image_to_video, so the exclusivity guidance is incomplete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.