Analyze audio
analyze_audioAnalyze an audio file. Modes:
transcript: Speech-to-text with word-level timestamps — returns
{language, full_text, words}where each word is{word, start_ms, end_ms, punctuated_word}. ⚠ MILLISECONDS here; get_clip'svoiceover_wordsgives the same words in SECONDS as{word, start, end}. Use for caption timing, script editing, or word-level visual sync. Works on narration and on sung lyrics: the provider is picked from the content, and if it returns nothing the other one is tried automatically. So an emptywordsmeans the audio genuinely has no intelligible speech, not that the wrong engine ran.silences: returns
{silences, total_silence_ms}, each span{start_ms, end_ms, peak_db}. Use for auto-trim, smart-split, or pause-aware editing.beats: returns
{tempo_bpm, beats_ms}— beats_ms is an array of MILLISECOND positions. Empty for non-music. Use to align animations/zooms to a beat.features: returns
{duration_ms, peak_db, rms_db, lufs, has_music, has_speech, speech_share}— loudness plus two INDEPENDENT hints —has_music(beats hold a steady period) andhas_speech(at least 40% of the audible seconds sound like talking;speech_shareis that share, 0-1). Instrumentals read false; so do sung vocals and narration under a loud bed, so when the words matter, use transcript. Cheap dispatcher — call once to pick a mode, or for duration_ms to pass as add_audio's source_duration.energy: loudness over time —
{samples_per_second, start_ms, db}, db one RMS dBFS value per 1/samples_per_second s from start_ms (default 4 a second; energy_options.samples_per_second 1-20). Read builds, dips and hits from it.sections: how a track is built —
{loud_db, sections: [{label, start_ms, end_ms, db, onsets_per_s}], drops: [{t_ms, rise_db, quiet_start_ms?, quiet_end_ms?}]}, label intro | build | drop | main | breakdown | outro; a drop's t_ms is the hit (within ~10 ms), quiet_* the dip just before it. Set animation cues on beats (beats mode) or drops.describe: what the audio IS —
{kind: speech|music|mixed, classification_confidence, duration_seconds, music?: {tempo_bpm, key, mood, genre, energy, instruments, description, suggested_search_query}, speech?: {...}}. Pass suggested_search_query to find(type='music') for a similar track. Speech is not transcribed here; use transcript for the words.
Source: provide exactly one of mcp_upload_id (from upload tool) or file_url (already-public URL, e.g. a find(type='music') result or Clueso CDN file). Optional time range: range_start_ms / range_end_ms crops the analysis window.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"mode": "<mode>"
},
"name": "analyze_audio"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | Which analysis to run. | |
| file_url | No | Public http(s) URL to the audio. Use when the file is already accessible (e.g. find(type='music') result, Clueso CDN). Validated against private/loopback ranges. | |
| range_end_ms | No | End of analysis window in milliseconds. Default: whole file. | |
| mcp_upload_id | No | mcp_upload_id from upload_file, or a Clueso upload_id. Use this OR file_url, not both. | |
| energy_options | No | Options for mode='energy'. | |
| range_start_ms | No | Start of analysis window in milliseconds. Default 0 (beginning). | |
| silences_options | No | Options for mode='silences'. | |
| transcript_options | No | Options for mode='transcript'. |