sonicmatch-mcp

Open-source MCP server that recommends license-safe background music the way Instagram Stories / Reels feel: drop footage, get a shortlist that already matches energy, then pick a 15s hook.

Source: js713-lab/sonic-match-mcp. The installable package and CLI are named sonicmatch-mcp.

This is infrastructure for editors and agents, not another music chatbot.

Video or URL in
  → scene / mood / pace / speech analysis
  → license-safe BGM shortlist
  + beat/cut hints
  + optional mix preview

Do not treat this as “script in → YouTube Music search out.” That already exists (mcp-bgm-recommender). Sonicmatch watches the video.

North star: ingest_video → analyze_video_music → recommend_bgm → preview_mix → export_mix_spec

License warning (read this)

• The code is MIT.

• Every track has its own license. It is printed on every recommendation.

• Nothing here is an official Instagram sticker, TikTok Commercial Music Library track, or YouTube Audio Library API result.

• Do not recommend commercial pop unless the adapter is explicitly a user-owned licensed library.

• CC-BY still needs attribution. CC-BY-NC is not ok for ads / shops. Content ID can still hit you if you point at the wrong source.

Quick start

Requires Python 3.10+ and ffmpeg / ffprobe on PATH. yt-dlp is optional and off by default (SONICMATCH_ALLOW_YTDLP=0) because platform extractors break and may violate ToS. Prefer a local file.

pip install git+https://github.com/js713-lab/sonic-match-mcp.git

# or from a clone
git clone https://github.com/js713-lab/sonic-match-mcp.git
cd sonic-match-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env   # optional keys

# stdio (Claude Desktop / Cursor)
sonicmatch-mcp

# streamable HTTP (web editors)
sonicmatch-mcp --http --port 8765

With uv:

uv venv && uv pip install -e ".[dev]"
uv run sonicmatch-mcp

v0.2 works offline-ish with the checked-in seed catalog. Gemini, Jamendo, and Freesound are optional and degrade with a note in the tool response.

# tests (generates tiny color mp4s with ffmpeg)
pytest

Example agent prompt

I dropped ./clip.mp4. Analyze it for an Instagram Reel and recommend 5 instrumental BGMs. Then mix the top pick with ducking and give me the ffmpeg command.

Claude Desktop

claude_desktop_config.json:

{
  "mcpServers": {
    "sonicmatch": {
      "command": "/absolute/path/to/sonicmatch-mcp/.venv/bin/sonicmatch-mcp",
      "args": [],
      "env": {
        "GEMINI_API_KEY": "",
        "JAMENDO_CLIENT_ID": "",
        "FREESOUND_API_KEY": ""
      }
    }
  }
}

Cursor

.cursor/mcp.json (project) or ~/.cursor/mcp.json:

{
  "mcpServers": {
    "sonicmatch": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/sonicmatch-mcp", "run", "sonicmatch-mcp"]
    }
  }
}

Copy-paste configs live in examples/claude_desktop.mcp.json and examples/cursor.mcp.json. User-owned Epidemic/Artlist JSON shape: examples/user_library.example.json. Registry metadata: server.json.

HTTP editors can point at http://127.0.0.1:8765/mcp after sonicmatch-mcp --http.

Architecture

flowchart TB
  subgraph mcp [MCP Server - FastMCP / Python - stdio + HTTP]
    tools[ingest_video / analyze_video_music / recommend_bgm / preview_mix / export_mix_spec / suggest_cuts]
  end
  tools --> ingest
  tools --> brain
  tools --> hub
  tools --> mixer
  ingest[Ingestor<br/>yt-dlp · ffmpeg · ffprobe · URL/file]
  brain[Video Brain<br/>Gemini / local VL · librosa · PySceneDetect · Whisper]
  hub[Music Hub<br/>seed CC · Jamendo · Freesound · user library · generate]
  mixer[Mixer<br/>ffmpeg · ducking · loop/trim · EDL cuts]
  hub --> index[Track index<br/>tags + license + embeddings · SQLite · optional LanceDB]

Hard rule: never send raw multi-MB video through the MCP payload. Store locally, pass an asset_id. Loopback, file://, and private IPs are rejected (SSRF).

MCP tools

Also ships a prompt template: “Score this video like an IG music sticker.”

Product rules (Instagram-like, not Instagram)

• Prefer instrumental when speech_coverage > 0.25

• Recommend a hook window, not the whole song

• Show why (cuts at 0.8s average, 112 BPM, warm gold hour)

• Always return license + attribution text

• 3–7 tracks, not 40

• User can override mood / genre / no-lyrics / platform / energy

• Never claim “cleared for Instagram official sticker” unless it actually is

VideoSonic profile

Analysis returns structured JSON, not a paragraph:

{
  "duration_sec": 18.4,
  "aspect": "9:16",
  "content_type": "lifestyle",
  "has_speech": true,
  "speech_coverage": 0.62,
  "existing_music": false,
  "overall_mood": ["warm", "playful"],
  "energy_mean": 0.62,
  "energy_curve": [{"t": 0, "energy": 0.3}, {"t": 4, "energy": 0.8}],
  "pacing": "fast-cut",
  "scenes": [{"start": 0, "end": 3.2, "description": "cafe exterior", "energy": 0.4}],
  "hook_window": [9.0, 15.0],
  "suggested_bpm": [95, 118],
  "avoid": ["dark cinematic drone", "aggressive trap", "lyrics-dense"],
  "search_queries": ["warm acoustic pop instrumental cafe"],
  "platform_hint": "instagram_reel",
  "analyzer": "local"
}

• Primary: Gemini video understanding when GEMINI_API_KEY is set.

• Fallback: ffmpeg scene cuts + WAV energy / silence / ZCR heuristics. Optional faster-whisper, scenedetect, librosa if installed (pip install 'sonicmatch-mcp[local-vl]').

Music hub

Pluggable, license-first. v0 ships:

Ranking (weighted): mood/energy → instrumental if speech → duration/loop → BPM vs cut rate → license fit → tag embedding cosine → user constraints.

Tracks are indexed in SQLite (~/.cache/sonicmatch-mcp/db/tracks.sqlite) with a 24-d tag embedding. If lancedb is installed (pip install 'sonicmatch-mcp[embeddings]'), vectors are also upserted there.

Seed tracks have no remote audio files on purpose (you should host files you actually have the rights to). preview_mix synthesizes a CC0 demo bed so the mixer still runs offline. generate_bed is a catalog-miss fallback and is not cleared for ads.

Docker

docker build -t sonicmatch-mcp .
docker run --rm -p 8765:8765 -v sonic-cache:/data/cache sonicmatch-mcp

Roadmap

• Freesound adapter (loops / beds)

• Tag embeddings in SQLite (+ optional LanceDB extra)

• Epidemic Sound / Artlist as user-owned JSON plugins (no scrape)

• Beat-grid vs scene-cut suggestions (EDL-ish suggest_cuts)

• MCP registry listing (server.json)

• Generate tool, marked source=generated (local demo; swap a real model at your own legal risk)

• Real CLAP audio embeddings

• Official MCP registry listing via GitHub Release MCPB (see PUBLISH.md)

• PyPI release

• Beat-grid auto-recut of the video itself (not just EDL hints)

Why this can be a good open-source project

Yes if you nail: (1) video-native analysis, (2) license honesty on every row, (3) editor-shaped output (hook in/out, ducking, mix spec).

No if you only wrap YouTube Music search. That is a weekend clone and a copyright magnet.

Day-1 risk gates (enforced in code, not slogans):

Use cases

IG Reel / Story · Shopee product clip · YouTube Shorts agent · CapCut/Premiere companion · campus recap · podcast clipper · travel-vlog batch · brand-kit lock (BPM + no vocals) · silent-film / accessibility · multi-agent studio.