mcp-ear
by mrtdemissek
README.md
# mcp-ear 👂
**An ear for AI assistants.** Language models like Claude can *talk* about sound, and some can even *generate* it — but most can't listen. `mcp-ear` is an [MCP](https://modelcontextprotocol.io) server that gives Claude (or any MCP client) the ability to listen to audio files on your computer:
- 🗣️ **Hear speech** — transcribe it with a local Whisper model (90+ languages, offline, free)
- 🎵 **Hear music** — tempo, beat, pitch range, melody notes, musical key
- 🔊 **Hear sound** — loudness, dynamics, silences, frequency balance, noise vs. tone
- 👁️ **See sound** — a spectrogram image, so a vision-capable model can literally look at what it sounds like
Everything runs on your own machine. No API keys, no audio leaves your computer.
[Türkçe README](README.tr.md)
## What it looks like
> **You:** What's in `C:\recordings\voice-note.wav`?
>
> **Claude:** *(calls `listen`)* It's a 7-second voice note: "Hello Claude. This is a test of your new ear. The quick brown fox jumps over the lazy dog." *(calls `spectrogram`)* The speaker has a fairly low, steady voice (around 180 Hz) with three clear phrases separated by short pauses.
The spectrogram tool returns an image like this (waveform on top; time → right, pitch ↑ up, brightness = loudness; cyan line = detected pitch):

## Tools
| Tool | What it does |
|---|---|
| `audio_info` | Duration, sample rate, channels, format |
| `listen` | Speech or sung lyrics → text with timestamps (optional per-word timing), auto language detection. `mode`: `speech`, `music` or `auto` (switches to music mode when the voice filter would discard most of a song). Known Whisper hallucinations ("Thanks for watching", subtitle credits) are filtered out |
| `analyze` | Loudness & dynamics, silences, tempo & beat steadiness, pitch range & common notes, key estimate, energy per frequency band, plain-language hints |
| `melody` | The note sequence (e.g. `C4 D4 E4 F4 G4`) with timing and tuning in cents |
| `spectrogram` | PNG image: waveform + mel spectrogram + pitch contour |
All tools take an absolute file `path`, and most accept `start` / `duration` (seconds) to focus on part of a long file.
**Formats:** WAV, FLAC, OGG and MP3 work out of the box. M4A, AAC, OPUS, WEBM and video files (MP4, MOV…) need [ffmpeg](https://ffmpeg.org/download.html) on your PATH.
## Install
You need Python 3.10+. The easiest way is [uv](https://docs.astral.sh/uv/getting-started/installation/), which installs and runs it in one step.
### Claude Code
```bash
claude mcp add ear -- uvx --from git+https://github.com/mrtdemissek/mcp-ear mcp-ear
```
### Claude Desktop
Open *Settings → Developer → Edit Config* and add this to `claude_desktop_config.json`:
```json
{
"mcpServers": {
"ear": {
"command": "uvx",
"args": ["--from", "git+https://github.com/mrtdemissek/mcp-ear", "mcp-ear"]
}
}
}
```
Restart Claude Desktop. Then ask it to listen to a file by its full path.
### Without uv
```bash
pip install git+https://github.com/mrtdemissek/mcp-ear
```
Then use `mcp-ear` as the command in the configs above (no `args`).
> **First run:** the dependencies are a few hundred MB, and the first `listen` call downloads the Whisper model (`small` ≈ 480 MB). After that it works offline.
## Settings (environment variables)
| Variable | Default | Meaning |
|---|---|---|
| `MCP_EAR_WHISPER_MODEL` | `small` | `tiny`, `base`, `small`, `medium`, `large-v3`, `large-v3-turbo`. Bigger = more accurate, slower. For songs and non-English speech, `medium` or larger is noticeably better |
| `MCP_EAR_DEVICE` | `auto` | `auto`, `cpu` or `cuda`. With `auto`, a GPU is used when available; if the CUDA libraries are missing it falls back to CPU by itself |
| `MCP_EAR_COMPUTE_TYPE` | `default` (`int8` on CPU) | Passed to faster-whisper |
In Claude Desktop, set them with an `"env": { "MCP_EAR_WHISPER_MODEL": "medium" }` block next to `"args"`.
## Honest limitations
- It **analyzes** sound; it doesn't *experience* it. The plain-language hints ("speech-like", "noise-like", "steady pulse") are heuristics, and the model is told to treat them as guesses.
- Sung lyrics are much harder than speech: expect misheard words, especially with the `small` model. In our test on a live acoustic Turkish song, `medium` got most lines right or nearly right but still misheard some words.
- `melody` works best on a single voice or instrument. On a full band mix you get the most prominent line, not a full transcription.
- Key and tempo estimates can be off by an octave (60 vs 120 BPM) or land on the relative key (C major vs A minor).
- It doesn't yet recognize sound *events* (dog bark, doorbell, applause), separate speakers, or listen to a live microphone. See the roadmap.
## Roadmap
- [ ] Sound event recognition (e.g. YAMNet/PANNs: "dog barking", "glass breaking", "applause")
- [ ] Speaker diarization ("who spoke when")
- [ ] Live microphone listening
- [ ] Compare two recordings (A/B)
- [ ] PyPI release (`uvx mcp-ear`)
Contributions are welcome. Open an issue or PR.
## Development
```bash
git clone https://github.com/mrtdemissek/mcp-ear
cd mcp-ear
python -m venv .venv
.venv/bin/pip install -e ".[dev]" # Windows: .venv\Scripts\pip
.venv/bin/pytest
```
The tests synthesize sounds with known answers (an A4 note, a C major scale, a 120 BPM click track, white noise) and check that the ear hears them correctly.
## How it works
- Audio decoding: [soundfile](https://github.com/bastibe/python-soundfile), with [ffmpeg](https://ffmpeg.org) as a fallback
- Speech: [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (OpenAI Whisper on CTranslate2)
- Music & signal analysis: [librosa](https://librosa.org) (pYIN pitch tracking, beat tracking, chroma, Krumhansl–Kessler key profiles)
- Images: [matplotlib](https://matplotlib.org)
- Protocol: the official [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk) (v1 and v2 both supported)
## License
MIT
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues