Skip to main content
Glama
io.github.js713-lab/sonicmatch-mcp

io.github.js713-lab/sonicmatch-mcp

by js713-lab

Sonicmatch

CI PyPI License: MIT MCP Registry Python 3.10+

Video-native MCP server for license-safe BGM. Watches the footage — not the script — and returns a shortlist, a 12–20s hook, and an ffmpeg ducking spec.

Package / CLI: sonicmatch-mcp. A Model Context Protocol server for Claude Desktop, Cursor, Codex, and other MCP clients. Drop an Instagram Reel, YouTube Short, or TikTok-style clip. Get royalty-free / Creative Commons matches with the license printed on every row.

Video-to-BGM already exists. The wedge is not “I also match music”:

  • it watches the footage, not the script

  • it returns a hook window + ffmpeg ducking spec

  • it is agent-native

  • it prints the license instead of lying

Catalog quality will kill or save this. More tools will not.

Video or URL in
  → scene / mood / pace / speech analysis
  → license-safe BGM shortlist
  + beat/cut hints
  + optional mix preview

Do not treat this as “script in → YouTube Music search out.” That already exists (mcp-bgm-recommender). Sonicmatch watches the video.

You own

You do not own

Local file / public URL ingest

Platform music licenses

Mood, energy curve, speech vs silence, scene cuts

Meta/TikTok “trending audio” graph

CC / royalty-free catalogs + optional paid adapters

Spotify / IG official libraries

Ranked tracks, preview URLs, mix spec, ffmpeg

Auto-publish to Instagram

North star: ingest_video → analyze_video_music → recommend_bgm → preview_mix → export_mix_spec

License warning (read this)

  • The code is MIT.

  • Every track has its own license. It is printed on every recommendation.

  • Nothing here is an official Instagram sticker, TikTok Commercial Music Library track, or YouTube Audio Library API result.

  • Do not recommend commercial pop unless the adapter is explicitly a user-owned licensed library.

  • CC-BY still needs attribution. CC-BY-NC is not ok for ads / shops. Non-commercial tracks are never auto-recommended.

  • For ads / shops, wire a user-owned Artlist / Epidemic JSON (examples/user_library.example.json). Do not scrape those sites.

  • Content ID can still hit you if you point at the wrong source. A CC label is not a waiver.

Related MCP server: Sonicmatch

Demo

I dropped ./clip.mp4. Analyze it for an Instagram Reel and recommend 5 instrumental BGMs. Then mix the top pick with ducking and give me the ffmpeg command.

That prompt is the product. Install below, wire Claude Desktop, Cursor, or Codex, and paste it.

Quick start

Requires Python 3.10+ and ffmpeg / ffprobe on PATH. yt-dlp is optional and off by default (SONICMATCH_ALLOW_YTDLP=0) because platform extractors break and may violate ToS. Prefer a local file.

The PyPI project is sonicmatch-mcp. The console script is sonicmatch-mcp, which is what uvx and pipx run. python -m sonicmatch is the same server. The PyPI badge stays empty until the first tag is published. Install from git until then. Maintainer steps: PUBLISH.md.

Works today (git)

uvx --from git+https://github.com/js713-lab/sonic-match-mcp.git sonicmatch-mcp
pip install git+https://github.com/js713-lab/sonic-match-mcp.git
sonicmatch-mcp --version   # or: python -m sonicmatch --version

After the first PyPI release

uvx sonicmatch-mcp
# or
pipx install sonicmatch-mcp
sonicmatch-mcp --version

From a clone (editable + tests)

git clone https://github.com/js713-lab/sonic-match-mcp.git
cd sonic-match-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env   # optional keys

# stdio (Claude Desktop / Cursor / Codex)
sonicmatch-mcp

# streamable HTTP (web editors). No authentication. Keep it on loopback.
sonicmatch-mcp --http --port 8765
# same clone, uv
uv venv && uv pip install -e ".[dev]"
uv run sonicmatch-mcp
pytest          # generates tiny color mp4s with ffmpeg
ruff check src tests

The seed catalog is about 20 Reel-editor rows (cafe, product, talking-head, travel, food, fashion, event) plus a vocal fixture and a CC-BY-NC fixture. It runs without API keys. Gemini, Jamendo, and Freesound are optional and degrade with a note in the tool response. Seed rows have no hosted audio on purpose — preview_mix synthesizes a demo bed. For real ads, point SONICMATCH_LIBRARY_PATH / EPIDEMIC_LIBRARY_PATH / ARTLIST_LIBRARY_PATH at JSON you already licensed.

Variable

Required

What it does

GEMINI_API_KEY

no

Gemini video understanding for analyze_video_music

JAMENDO_CLIENT_ID

no

Jamendo search

FREESOUND_API_KEY

no

Freesound beds and loops

SONICMATCH_LIBRARY_PATH

no

User-owned library JSON. Also EPIDEMIC_LIBRARY_PATH, ARTLIST_LIBRARY_PATH

SONICMATCH_ALLOW_YTDLP

no

1 turns on platform URL ingest. Default 0

SONICMATCH_MAX_DOWNLOAD_MB

no

Download cap. Default 200

SONICMATCH_HTTP_HOST / SONICMATCH_HTTP_PORT

no

--http bind. Default 127.0.0.1:8765

Features

  • Watches the picture. Gemini video understanding when GEMINI_API_KEY is set; otherwise local ffmpeg / audio heuristics (optional Whisper, PySceneDetect, librosa).

  • License on every row. CC / royalty-free / user-owned library. Non-commercial tracks are never auto-recommended. Generated beds stay out of auto recs.

  • Editor-shaped output. 12–20s hook in/out, speech ducking, ffmpeg recipe, mix spec for CapCut / Premiere / DaVinci / your agent.

  • Agent-native MCP. stdio for Claude Desktop and Cursor; streamable HTTP for web editors. Never ships raw multi-MB video through the payload — you get an asset_id.

  • Works without API keys. 20-track seed catalog for Reels. Optional Jamendo, Freesound, and user-owned Artlist / Epidemic JSON.

  • Cuts on the beat. suggest_cuts snaps scene cuts to a BPM grid and returns EDL-ish intro / peak / outro.

  • Brand lock. Save BPM / mood / no-vocals kits and pass brand_kit=… into recommend_bgm.

  • Series, not one-offs. analyze_batch (max 20) clusters mood and returns one shared mini-playlist.

  • Honest about closed graphs. No fake “trending audio,” official IG stickers, or Content-ID-safe stamps. yt-dlp platform ingest is opt-in.

Tools

Tool

What it does

status

ffmpeg / keys / seed count / day-1 risk gates

ingest_video

Local path or HTTPS URL → asset_id (never video bytes). Platform URLs need SONICMATCH_ALLOW_YTDLP=1

analyze_video_music

Mood, energy curve, speech, scenes, hook window, BPM, search queries

recommend_bgm

3–7 ranked tracks + why + license + hook in/out

search_music

Free-text / BPM / mood over seed + optional catalogs

get_track

One track’s metadata, license, attribution, URLs

preview_mix

Hook trim, loop, optional ducking → preview files + ffmpeg + mix spec

export_mix_spec

Mix spec + ffmpeg + attribution (no render unless render=true)

suggest_cuts

Beat grid, snapped scene cuts, EDL, intro / peak / outro

generate_bed

Demo bed marked source=generated. Requires i_understand_not_commercially_cleared=true

save_brand_kit

Persist BPM / moods / no-vocals for recommend_bgm(brand_kit=…)

analyze_batch

Up to 20 clips → mood cluster + shared mini-playlist

Also ships a prompt template: “Score this video like an IG music sticker.”

Product rules (Instagram-like, not Instagram)

  • Prefer instrumental when speech_coverage > 0.25

  • Recommend a hook window, not the whole song

  • Show why (cuts at 0.8s average, 112 BPM, warm gold hour)

  • Always return license + attribution text

  • 3–7 tracks, not 40

  • User can override mood / genre / no-lyrics / platform / energy

  • Never claim “cleared for Instagram official sticker” unless it actually is

Example

Prompt:

I dropped ./clip.mp4. Analyze it for an Instagram Reel and recommend 5 instrumental BGMs. Then mix the top pick with ducking and give me the ffmpeg command.

The agent should call, in order:

  1. ingest_video with the local path. The result is an asset_id, not video bytes.

  2. analyze_video_music with platform_hint set to instagram_reel.

  3. recommend_bgm with that asset_id and max_results of 5.

  4. preview_mix on the top track with ducking, then export_mix_spec.

A recommendation row carries score, reason, suggested_in_out (the hook), ducking, license_ok_for_platform, and content_id_risk (unknown or likely, never cleared). Seed fixtures include seed_golden_hour_latte (CC0-1.0) and seed_indie_porch_110 (CC-BY-4.0). They have no hosted audio. preview_mix synthesizes a demo bed. CC-BY-NC rows are searchable and are not auto-recommended.

Claude Desktop

claude_desktop_config.json. From git, no clone:

{
  "mcpServers": {
    "sonicmatch": {
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/js713-lab/sonic-match-mcp.git",
        "sonicmatch-mcp"
      ],
      "env": {
        "GEMINI_API_KEY": "",
        "JAMENDO_CLIENT_ID": "",
        "FREESOUND_API_KEY": ""
      }
    }
  }
}

After a clone, command can be the absolute path of .venv/bin/sonicmatch-mcp. After the PyPI release, args is just ["sonicmatch-mcp"]. Copy-paste files: examples/claude_desktop.mcp.json, examples/claude_desktop.pypi.mcp.json.

Cursor

.cursor/mcp.json (project) or ~/.cursor/mcp.json. From git, no clone — examples/cursor.mcp.json:

{
  "mcpServers": {
    "sonicmatch": {
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/js713-lab/sonic-match-mcp.git",
        "sonicmatch-mcp"
      ],
      "env": {
        "GEMINI_API_KEY": "",
        "JAMENDO_CLIENT_ID": "",
        "FREESOUND_API_KEY": ""
      }
    }
  }
}

After PyPI, use examples/cursor.pypi.mcp.json (uvx with args ["sonicmatch-mcp"]). From a clone: "command": "uv", "args": ["--directory", "/absolute/path/to/sonic-match-mcp", "run", "sonicmatch-mcp"]. After pip install, "command": "python3", "args": ["-m", "sonicmatch"] works if that interpreter has the package.

Codex

Codex reads ~/.codex/config.toml. The same file is shared with the Codex IDE extension and the ChatGPT desktop app. Works today — examples/codex.config.toml:

[mcp_servers.sonicmatch]
command = "uvx"
args = ["--from", "git+https://github.com/js713-lab/sonic-match-mcp.git", "sonicmatch-mcp"]

[mcp_servers.sonicmatch.env]
GEMINI_API_KEY = ""
JAMENDO_CLIENT_ID = ""
FREESOUND_API_KEY = ""

Or from the CLI:

codex mcp add sonicmatch -- uvx --from git+https://github.com/js713-lab/sonic-match-mcp.git sonicmatch-mcp

After the PyPI release, the args shrink to sonicmatch-mcp. That file is examples/codex.pypi.config.toml:

codex mcp add sonicmatch -- uvx sonicmatch-mcp

codex mcp list should show sonicmatch. A trusted project can use .codex/config.toml instead of the home file.

User-owned Epidemic/Artlist JSON shape: examples/user_library.example.json. Registry metadata: server.json. HTTP editors can point at http://127.0.0.1:8765/mcp after sonicmatch-mcp --http.

Hosted (sonic.codecrafter.dev)

The GitHub repo stays MIT. Running it yourself is not metered.

The public endpoint charges credits per tool call. A trial key is 25 credits. Pack prices: Reel 500 credits / $9, Studio 3000 credits / $39. Card checkout is not connected on the host yet.

The landing page is https://sonic.codecrafter.dev. Use Use the sample clip there, or download examples/sample-clip.mp4 and upload it, to try a match without your own footage.

  1. Open https://sonic.codecrafter.dev and take a trial key (or use a paid key).

  2. export SONICMATCH_API_KEY=sm_…

  3. Add to Cursor: the button on that page, or paste examples/cursor.hosted.mcp.json into ~/.cursor/mcp.json.

{
  "mcpServers": {
    "sonicmatch": {
      "url": "https://sonic.codecrafter.dev/mcp",
      "headers": {
        "Authorization": "Bearer ${env:SONICMATCH_API_KEY}"
      }
    }
  }
}

--http has no authentication. Keep it on loopback. The Docker image binds 0.0.0.0 so the container port works — do not publish that port to the internet. See SECURITY.md.

Architecture

flowchart TB
  subgraph mcp [MCP Server - FastMCP / Python - stdio + HTTP]
    tools[ingest_video / analyze_video_music / recommend_bgm / preview_mix / export_mix_spec / suggest_cuts]
  end
  tools --> ingest
  tools --> brain
  tools --> hub
  tools --> mixer
  ingest[Ingestor<br/>yt-dlp · ffmpeg · ffprobe · URL/file]
  brain[Video Brain<br/>Gemini / local VL · librosa · PySceneDetect · Whisper]
  hub[Music Hub<br/>seed CC · Jamendo · Freesound · user library · generate]
  mixer[Mixer<br/>ffmpeg · ducking · loop/trim · EDL cuts]
  hub --> index[Track index<br/>tags + license + embeddings · SQLite · optional LanceDB]

Hard rule: never send raw multi-MB video through the MCP payload. Store locally, pass an asset_id. Loopback, file://, and private IPs are rejected (SSRF).

VideoSonic profile

Analysis returns structured JSON, not a paragraph:

{
  "duration_sec": 18.4,
  "aspect": "9:16",
  "content_type": "lifestyle",
  "has_speech": true,
  "speech_coverage": 0.62,
  "existing_music": false,
  "overall_mood": ["warm", "playful"],
  "energy_mean": 0.62,
  "energy_curve": [{"t": 0, "energy": 0.3}, {"t": 4, "energy": 0.8}],
  "pacing": "fast-cut",
  "scenes": [{"start": 0, "end": 3.2, "description": "cafe exterior", "energy": 0.4}],
  "hook_window": [9.0, 15.0],
  "suggested_bpm": [95, 118],
  "avoid": ["dark cinematic drone", "aggressive trap", "lyrics-dense"],
  "search_queries": ["warm acoustic pop instrumental cafe"],
  "platform_hint": "instagram_reel",
  "analyzer": "local"
}
  • Primary: Gemini video understanding when GEMINI_API_KEY is set.

  • Fallback: ffmpeg scene cuts + WAV energy / silence / ZCR heuristics. Optional faster-whisper, scenedetect, librosa if installed (pip install 'sonicmatch-mcp[local-vl]').

Music hub

Pluggable, license-first. v0 ships:

Adapter

When

License reality

Seed catalog (data/seed_tracks.json)

always

20 CC0 / CC-BY Reel beds + a vocal fixture + a CC-BY-NC fixture (NC is never auto-recommended)

Jamendo

JAMENDO_CLIENT_ID

CC, check commercial

Freesound

FREESOUND_API_KEY

CC, good for beds/loops not songs

User library JSON

SONICMATCH_LIBRARY_PATH / EPIDEMIC_LIBRARY_PATH / ARTLIST_LIBRARY_PATH

you already licensed it; we do not scrape paid sites

Generate

generate_bed

always source=generated; local sine demo unless you swap a real model

Ranking (weighted): mood/energy → instrumental if speech → duration/loop → BPM vs cut rate → license fit → tag embedding cosine → user constraints. recommend_bgm drops non-commercial and generated tracks instead of downranking them.

Tracks are indexed in SQLite (~/.cache/sonicmatch-mcp/db/tracks.sqlite) with a 24-d tag embedding. If lancedb is installed (pip install 'sonicmatch-mcp[embeddings]'), vectors are also upserted there.

Seed tracks have no remote audio files on purpose (you should host files you actually have the rights to). preview_mix synthesizes a CC0 demo bed so the mixer still runs offline. generate_bed is a catalog-miss fallback and is not cleared for ads.

Docker

docker build -t sonicmatch-mcp .
# Loopback-only publish. The process inside the container has no HTTP auth.
docker run --rm -p 127.0.0.1:8765:8765 -v sonic-cache:/data/cache sonicmatch-mcp

Roadmap

Catalog > new tools.

  • Freesound adapter (loops / beds)

  • Tag embeddings in SQLite (+ optional LanceDB extra)

  • Epidemic Sound / Artlist as user-owned JSON plugins (no scrape)

  • Beat-grid vs scene-cut suggestions (EDL-ish suggest_cuts)

  • MCP registry listing (server.json)

  • Generate tool, marked source=generated (local demo; swap a real model at your own legal risk)

  • Official MCP registry listing via GitHub Release MCPB (see PUBLISH.md)

  • Non-commercial licenses excluded from auto recommend_bgm

  • 20 seed beds a Reel editor would actually keep, with audio you host

  • User-owned Artlist / Epidemic JSON as the default path for ads

  • Real CLAP audio embeddings

  • PyPI project metadata, console script, and tag-triggered Trusted Publishing (.github/workflows/release.yml)

  • First upload on PyPI (uvx sonicmatch-mcp). Maintainer steps are in PUBLISH.md. The name was unregistered as of 2026-10-21.

Why this can be a good open-source project

Yes if you nail: (1) video-native analysis, (2) license honesty on every row, (3) editor-shaped output (hook in/out, ducking, mix spec), (4) a catalog someone would keep.

No if you only wrap YouTube Music search, or if the first five recs sound like leftover stock beds.

Day-1 risk gates (enforced in code, not slogans):

Risk

Gate

Content ID

Every rec/search/get_track includes content_id_warning. CC/RF is never "Content-ID-safe". content_id_risk is unknown or likely, never cleared.

yt-dlp ToS / broken extractors

Platform URL ingest is off unless SONICMATCH_ALLOW_YTDLP=1. Failures map to YTDLP_EXTRACTOR and tell you to pass a local file.

Upload size / SSRF

HTTPS-only remote ingest, no file:// / loopback / private IPs, SONICMATCH_MAX_DOWNLOAD_MB (default 200) on files, HTTP, and yt-dlp --max-filesize.

“Trending” is a closed Meta graph

Queries for trending/viral/IG audio/TikTok sound return empty + TRENDING_UNAVAILABLE. recommend_bgm always sets trending_available=false.

Generation-model commercial terms

generate_bed refuses unless i_understand_not_commercially_cleared=true. Generated tracks are excluded from auto recommend_bgm.

Use cases

IG Reel / Story · Shopee product clip · YouTube Shorts agent · CapCut/Premiere companion · campus recap · podcast clipper · travel-vlog batch · brand-kit lock (BPM + no vocals) · silent-film / accessibility · multi-agent studio.

Contributing

CONTRIBUTING.md · CODE_OF_CONDUCT.md · SECURITY.md · CHANGELOG.md · starter issues

License

MIT. Track licenses are independent of the repo license.

From CodeCrafter.

Available Tools

12 tools
analyze_batchA

Analyze up to 20 clips, cluster moods, return one consistent mini-playlist for the series.

Inputs: sources (required), extra_notes, max_seconds, platform_hint. Returns: object

analyze_video_musicA

Produce a structured VideoSonic profile for BGM matching.

platform_hint: instagram_story | instagram_reel | tiktok | youtube_short | youtube_long | generic.

Uses Gemini video understanding when GEMINI_API_KEY is set; otherwise local ffmpeg/audio heuristics (and optional faster-whisper / PySceneDetect if installed). Always fills search_queries even if analysis is weak.

extra_notes: optional caption/script the user already has — used as a hint, not as a replacement for watching the video.

Inputs: asset_id (required), extra_notes, platform_hint. Returns: object

export_mix_specB

Export the mix spec + ffmpeg recipe + attribution without requiring a render.

This is the editor companion output (CapCut / Premiere / DaVinci / agent). Set render=true to also write preview files (same as preview_mix).

Inputs: asset_id (required), track_id (required), bgm_db, render, +2 more. Returns: object

generate_bedA

Generate a bed when the catalog misses. Always marked source=generated.

Refuses unless i_understand_not_commercially_cleared=true. Local fallback is a sine-tremolo demo. NOT cleared for ads. Check Suno/Stable Audio/Lyria terms before swapping in a real generator. Generated tracks are excluded from recommend_bgm auto catalogs.

Inputs: prompt (required), bpm, energy, asset_id, +2 more. Returns: object

get_trackC

Return one track's metadata, license string, attribution text, and URLs.

Inputs: track_id (required). Returns: object

ingest_videoA

Ingest a local video path or public http(s) URL.

Accepts a filesystem path or an HTTPS video URL. YouTube/TikTok/Instagram via yt-dlp is OFF unless SONICMATCH_ALLOW_YTDLP=1 (ToS + extractor risk). Rejects file://, http, loopback, and private IPs (SSRF). Size-capped.

Extracts duration/fps/aspect with ffprobe, a 16 kHz mono wav, up to 12 scene keyframes, and a 360p proxy. Returns an asset_id. Never returns video bytes.

Call this first. Then call analyze_video_music with the asset_id.

Inputs: source (required), max_seconds. Returns: object

preview_mixA

Mix BGM under the video: hook trim, loop if needed, optional speech ducking.

Returns preview mp3 + low-res mp4 paths, the ffmpeg command, and a mix_spec JSON an editor (CapCut / Premiere / DaVinci / your agent) can re-apply.

Ducking uses ffmpeg sidechaincompress when the video has speech. Seed tracks without audio files get a synthesized CC0 demo bed so the pipeline still runs offline.

Inputs: asset_id (required), track_id (required), bgm_db, ducking, +1 more. Returns: object

recommend_bgmA

Rank license-safe BGM. Pass asset_id and/or a VideoSonic profile.

Defaults instrumental_only=True when speech_coverage > 0.25. catalog: auto | seed | jamendo | freesound | library auto = seed always, plus Jamendo/Freesound/user-library when configured.

Each recommendation includes score, reason, suggested song in/out (12–20s high-energy slice), ducking hint, and license_ok_for_platform. max_results is clamped to 3–7. brand_kit: name previously saved with save_brand_kit.

Inputs: mood, genre, catalog, profile, +5 more. Returns: object

save_brand_kitB

Save a brand kit (BPM lock + no vocals + mood whitelist). Pass brand_kit=name to recommend_bgm.

Inputs: name (required), avoid, moods, notes, +4 more. Returns: object

search_musicC

Free-text search over seed + optional Jamendo / Freesound / user library.

Inputs: query (required), limit, bpm_max, bpm_min, +4 more. Returns: object

statusA

Show ffmpeg/yt-dlp availability and which music adapters have keys.

Inputs: no named parameters. Returns: object

suggest_cutsA

Snap detected scene cuts onto a beat grid. Returns EDL-ish spans and intro/peak/outro acts.

bpm defaults to the VideoSonic suggested_bpm midpoint, or the track BPM if track_id is set.

Inputs: asset_id (required), bpm, track_id. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 12 tools

    Disambiguation4/5

    Most tools have clearly distinct roles in the pipeline (ingest, analyze, recommend, search, render, export, cuts, bed generation). However, preview_mix and export_mix_spec overlap significantly since export_mix_spec with render=true is explicitly 'same as preview_mix', and recommend_bgm vs search_music both perform music lookup with some boundary overlap.

    Naming Consistency4/5

    All tools follow the snake_case verb_noun pattern (ingest_video, recommend_bgm, suggest_cuts, save_brand_kit, etc.), except for the noun-only 'status'. That minor inconsistency is the only deviation, making the set predictable.

    Tool Count5/5

    12 tools map directly to a coherent music-video workflow: video ingest, analysis, BGM recommendation/search, track details, mixing, export, cutting, generated beds, brand kit, and batch analysis. Every tool has a clear purpose and the count fits squarely in the well-scoped range.

    Completeness4/5

    The toolkit covers the essential lifecycle from video input to analysis, music selection, mix preview, specification export, and batch operation. Minor gaps exist like management of saved assets (e.g., updating brand kits or deleting generated beds), but the core workflow has no dead ends and agents can complete a full pipeline.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers