Chat with Audio

Part of Agentic Production.

Chat-driven audio enhancement: talk to your local Claude (Desktop or Claude Code) about a recording and let the toolkit analyze and improve it. A local A/B viewer lets you compare original and result in perfect sync — hearing and seeing.

Claude (chat)  ── MCP (stdio) ──>  Python orchestration ──> C++ DSP core (pybind11)
                                        │                    gate · compressor · limiter · EQ
                                        ├─> AI denoise (DeepFilterNet, optional)
                                        ├─> analysis (LUFS, SNR, hum, clipping, spectrum)
                                        └─> sessions ──> A/B viewer (http://127.0.0.1:8471)

What can you ask?

• "Analyze this file: /path/to/recording.wav" — metrics, scores and issues.

• "Make this sound better" — auto-improve: the tool detects speech/music and picks a chain by itself (highpass, hum notches, noise reduction, gate, EQ, compression, loudness), with an explanation for every step.

• "Reduce the noise" — noise reduction only (spectral gating or DeepFilterNet AI).

• "Bring the level up without clipping" — loudness normalization (BS.1770) with a true-peak limiter.

• "Cut 3 dB around 300 Hz and compress lightly" — an explicit chain via apply_chain.

• "Put the speech at −6 and balance the music" — refine_audio: segments speech/music/silence and runs a measure-and-adjust loop until the targets are right to the decibel. AI denoising is applied only when the speech SNR is low AND Whisper confirms intelligibility doesn't drop; the report includes the measurement history, the decisions and a final word-retention check.

• "Fix it only where something is wrong" — smart_edit: AI finds problem regions on the timeline — mains hum that comes and goes, noise that rises temporarily (AC, traffic), clusters of clipping, a passing low-frequency rumble — and applies a targeted mini-chain per region with crossfades. Everything outside the regions stays bit-for-bit untouched; the region map shows up as a timeline in the viewer. The surgical counterpart to improve_audio.

• "Save this as a preset" / "Do this like my podcast preset" — save_recipe, apply_recipe and list_recipes: keep the chain of a session that sounded right as a named recipe, reuse it on new files, and share it — a recipe is a small JSON file, and apply_recipe also accepts a path to someone else's recipe. Curated presets ship built in.

• "Make it even better, take your time" — optimize_audio: runs multiple pipeline variants (EQ, leveler, compressor, ClearVoice dereverberation) and lets the best one win on an objective score: Whisper word retention/confidence plus target deviation. The ranking comes back in the chat.

• "Transcribe this" — transcribe_audio (Whisper, [asr] extra).

• "Fix the clips and clicks" — repair_audio: declip (waveform reconstruction) and declick; improve_audio applies declip automatically when clipping is detected.

• "Make it sound like this reference" — match_reference: 1/3-octave match EQ + loudness match; keeps episodes/recording days consistent.

• "Split the stems" / "vocals up 3 dB" / "make a karaoke version" — separate_stems and rebalance_music (Demucs, [stems] extra).

• "Do the whole folder" — improve_folder: batch processing (improve/refine/optimize).

• "Show me what changed" — view_audio: a perceptual panel (auditory-scale spectrograms + difference map + level curves) that the AI inspects itself to judge what is audible.

• "This sounds good / this sounds bad" — rate_audio: train your own taste model; analyze_audio then scores new audio against your taste.

• "Set this up in Audition" — export_to_audition: stems + a .sesx multitrack session, opened directly in Adobe Audition.

• "Is this broadcast-proof?" — check_compliance: pass/fail report against EBU R128, ATSC A/85, Netflix 2.0 & 5.1 (dialogue-gated, on 5.1 detected on the center channel; LFE excluded from loudness), Apple Podcasts, Spotify, YouTube or ACX audiobook, including the technical QC gates (clipping, dropouts, dead channel, anti-phase, ITU-downmix peak) and ADM BWF (Dolby Atmos) recognition.

• "Master this for European broadcast, 48 kHz 24-bit" — master_for: masters to the spec (dialogue-gated specs steer on the detected speech), re-verifies, and exports a delivery file with high-quality SRC and bit-depth control. The compliance report shows up as a panel in the viewer.

• "Polish the dialogue" — the film-post steps: breath_control (dim breaths, never cut), deplosive (p/b-pops fixed on the pop only) and duck_music (music beds ride down under the speech level) — bundled in the built-in dialogue-polish recipe.

• "Give me the region map as markers" — export_markers: what the AI found lands as Audition marker CSV / Audacity labels in your DAW.

• "Fill the gaps with room tone" — fill_room_tone: digital holes (dropouts, edit gaps) get filled with the file's own ambience — the dialogue editor's classic.

• "There's a chair squeak at 12.3" — spectral_repair: RX-style spectral painting; the damaged time-frequency patch is repainted from its context while the programme underneath runs straight through.

• "Duck the music under the speech" — duck_music(mode="stems"): real sidechain ducking for music playing under dialogue (Demucs separation, vocals-driven gain, no pumping).

• "QC this file" / "QC this folder" — qc_report (one printable QC sheet) and qc_folder (a whole directory audited into one index table with verdicts).

• "Sync these recorders" — sync_tracks: the 32-track recorder. Files from different devices (lav, boom, field recorder, camera, phone) are aligned on the audio itself — sample-accurate GCC-PHAT, confidence per track, optional clock-drift correction — and come out as aligned WAVs plus an Audition session. A/B in the viewer: unsynced echo soup vs the synced sum.

• "Open the viewer" / "What exactly changed?" — A/B comparison; Claude reads the same session data the viewer shows.

Related MCP server: StemSplit Vocal Remover & Stem Separator

Documentation

Extensive docs live in docs/: getting started, the full tool reference, a workflows cookbook (podcast, broadcast delivery, film dialogue, restoration, QC), delivery compliance, smart regions, recipes and the architecture.

Installation (macOS)

Requires: uv, ffmpeg (brew install ffmpeg), Xcode Command Line Tools. Python 3.11 is fetched by uv itself.

cd chat-with-audio
uv sync --all-extras        # builds the C++ core and installs everything (incl. AI denoise)
uv run pytest               # 109 tests
uv run python scripts/mcp_smoke.py   # MCP smoke test

uv sync (without --all-extras) installs the basics without torch/DeepFilterNet; the tool then falls back to spectral gating automatically.

Registering with Claude (and Codex)

• Claude Code: lives in .mcp.json in the project folder (works automatically in this folder).

• Claude Desktop: entry chat-with-audio in ~/Library/Application Support/Claude/claude_desktop_config.json. Restart Claude Desktop after installing; the tools appear under "chat-with-audio".

• Codex CLI/app: registered as a global MCP server via codex mcp add chat-with-audio -- <uv-path> run --directory <project-folder> chat-with-audio-mcp (verify with codex mcp list). Same 30 tools, same sessions and viewer.

Note: run uv sync --all-extras first, otherwise the first server start may time out while building/downloading.

The viewer

open_viewer (or uv run ait viewer) starts it at http://127.0.0.1:8471. Space = play, a/b = switch between original and processed, r = residual (you hear exactly what the processing changed — ideal for artifact checking) while everything keeps playing in sync. Click the waveform to seek. Change the port with the AIT_VIEWER_PORT environment variable.

Under the waveforms sits a timeline: a content lane (speech/music/silence) and, for smart_edit sessions, an interventions lane showing exactly where which problem was treated — click a region to jump there and use r to hear what was removed.

Sessions live in ~/AudioImprove/sessions/ (override: AIT_SESSIONS_DIR), each with the original, result, analyses, chain + rationale, timeline, waveforms and spectrograms.

Recipes

A recipe is a saved processing chain as a small JSON file — reusable and shareable. Built-in presets (distilled from real sessions) ship with the package; your own live in ~/AudioImprove/recipes/ (override: AIT_RECIPES_DIR). Say "save this as my podcast preset" after a session that sounded right, apply it to new files with "do this like my podcast preset", and share the JSON file with anyone — apply_recipe also takes a file path. Steps are validated before anything runs.

Windows

1. Install uv, ffmpeg (winget install ffmpeg) and the Visual Studio Build Tools (C++ workload) for the native DSP core. Everything works without the Build Tools too, via the pure-Python fallback — in that case don't remove the C++ build step, it simply fails soft.

2. uv sync --all-extras in the project folder (DeepFilterNet ships win_amd64 wheels).

3. Register in %APPDATA%\Claude\claude_desktop_config.json with the full path to uv.exe and the project folder (same shape as the .mcp.json here).

Architecture

Loudness targets: speech −16 LUFS / TP −1.5 dBTP, music −14 LUFS / TP −1.0 dBTP.

Version pins (deliberate)

• Python 3.11 and numpy < 2.0: DeepFilterNet 0.5.x only ships wheels up to cp311 and requires numpy 1.x.

• torch/torchaudio < 2.9: DeepFilterNet imports torchaudio.backend, which was removed in torchaudio 2.9.

After changing the C++ code: uv sync --reinstall-package chat-with-audio.