# tone-listener-mcp

An MCP server that lets AI agents *listen* to guitar tone recordings and compare
a candidate tone against a reference. Use it to guide relative gain, EQ and effects
adjustments in software such as Soundshed Guitar, not to guess exact equipment.

Local NumPy/SciPy measurements handle levels, spectrum, dynamics and stereo.
Optional Gemini audio understanding adds cautious descriptions of what is audible.
Without an API key, analysis and comparison still return local measurements.

## Install

Requires Python **3.11 or newer**. Run from this repository:

```sh
python -m venv .venv
.venv/bin/pip install -e '.[dev]'
```

Runtime dependencies are MCP, NumPy, SciPy, SoundFile and google-genai, plus
Pydantic for response validation and HTTPX for typed provider-error handling.
No librosa or ffmpeg is required. SoundFile wheels bundle libsndfile, including
MP3 support. WAV, FLAC, OGG and MP3 are supported when the installed decoder can
read them. Convert M4A/AAC to WAV with your DAW or another converter first.

## Environment

| Variable | Default | Purpose |
|---|---|---|
| `GEMINI_API_KEY` | Unset | Enable Gemini uploads and listening; omit for local-only operation |
| `TONE_LISTENER_MODEL` | `gemini-3.8-flash` | Audio-capable Gemini model available to your account |
| `TONE_LISTENER_MAX_SECONDS` | `600` | Maximum selected duration; positive finite seconds |

`.env.example` is a reference, **not automatically loaded**. Export variables in
your shell or set them in your MCP client's environment configuration. Never
commit a real key. Model availability, API access and billing depend on your
Google account. The default follows the current [Gemini audio documentation](https://ai.google.dev/gemini-api/docs/audio);
the backend uses the SDK Files upload and `models.generate_content` APIs.

```sh
export GEMINI_API_KEY="your-key"
.venv/bin/tone-listener-mcp
```

The server speaks MCP over stdio, not an interactive terminal interface.

## Register with an MCP client

Generic `mcpServers` configuration (replace the absolute path):

```json
{
  "mcpServers": {
    "tone-listener": {
      "command": "/absolute/path/to/tone-listener-mcp/.venv/bin/tone-listener-mcp",
      "env": {
        "GEMINI_API_KEY": "your-key"
      }
    }
  }
}
```

VS Code `.vscode/mcp.json` uses `servers` and supports a secret input:

```json
{
  "inputs": [
    {
      "type": "promptString",
      "id": "gemini-key",
      "description": "Gemini API key",
      "password": true
    }
  ],
  "servers": {
    "tone-listener": {
      "type": "stdio",
      "command": "/absolute/path/to/tone-listener-mcp/.venv/bin/tone-listener-mcp",
      "env": {
        "GEMINI_API_KEY": "${input:gemini-key}"
      }
    }
  }
}
```

For offline use, omit `GEMINI_API_KEY` and the VS Code secret input. Paths to audio
are resolved on the machine running this server, not necessarily the client machine.

## Tools

All tools return structured JSON as well as MCP-compatible content. Invalid files
or intervals return an MCP tool error. Times are seconds; `start` defaults to zero,
`end` is exclusive and defaults to EOF. An endpoint past EOF is clamped to EOF.
`~` is expanded, and original channels/sample rate are retained for local metrics.
Only mono/stereo audio up to 192 kHz is accepted. Selected float32 decoded audio is
limited to 128 MiB in addition to the duration cap; shorten clips when either
limit is exceeded.

| Tool | Arguments | Result |
|---|---|---|
| `server_info` | None | Server name and version |
| `audio_metrics` | `path`, optional `start`, `end` | Local clip metadata, level, spectrum and stereo metrics; never uploads |
| `analyze_audio` | `path`, optional `start`, `end`, `question` | Local `metrics` plus Gemini `listening`, or `listening_unavailable` |
| `compare_audio` | `path_a`, `path_b`, optional `start_a`, `end_a`, `start_b`, `end_b`, `focus` | Reference A vs candidate B, RMS matching, deltas, EQ directions and optional A/B listening |

Example requests to an agent's MCP tool interface:

```json
{"path": "/recordings/reference.wav", "start": 12, "end": 20}
```

```json
{
  "path_a": "/recordings/reference.wav",
  "path_b": "/recordings/candidate.wav",
  "focus": "How should I change B's brightness and drive to approach A?"
}
```

### Local metrics

For a 0.5-peak, 1 kHz synthetic sine, selected fields look like this
(values rounded; this is an excerpt, not the complete response):

```json
{
  "level": {
    "peak_dbfs": -6.02,
    "rms_dbfs": -9.03,
    "crest_factor_db": 3.01,
    "short_term_rms_range_db": 0.0,
    "silent_window_fraction": 0.0
  },
  "spectrum": {
    "centroid_hz": 1000.0,
    "rolloff_85_hz": 1007.81,
    "bands": {
      "800_2000": {"available": true, "relative_db": 0.0}
    }
  },
  "stereo": null
}
```

Levels include 50 ms short-term RMS power percentiles (p10, p50, p90) and range
as dynamics proxies. These are **not LUFS** or a perceptual loudness measurement.
Spectrum uses Welch's method with a Hann window and up to 4096 frames, averaging
channel powers so antiphase stereo does not disappear. Frequency resolution and
Nyquist are reported. Band IDs cover `<80`, `80-250`, `250-800`, `800-2000`,
`2000-5000`, `5000-8000` and `>8000` Hz. `relative_db` is band power relative
to total spectral power, not an absolute EQ control value.

Stereo reports L/R correlation and side/mid energy ratio dB, with separate mid
and side RMS levels. `null` means undefined or unavailable, including silence,
mono stereo metrics, zero side/mid energy, or bands above Nyquist. A short-term
range containing a zero-energy window is undefined; inspect
`silent_window_fraction` as well. No NaN or Infinity is returned.

### Listening

`analyze_audio` returns a `listening` object with `summary`, `gain_level`,
`brightness`, `low_end`, `mids_character`, `pick_attack`, `compression_feel`,
`effects` (reverb/delay/modulation presence, amount and notes), `playing_notes`,
timestamped `evidence`, `uncertainties` and `confidence`.
Gain categories include clean, edge-of-breakup, crunch, overdrive, distortion,
high-gain and uncertain. Evidence timestamps are relative to the selected clip.

No key gives an explicit local-only response:

```json
{
  "model": "gemini-3.8-flash",
  "listening": null,
  "listening_unavailable": "GEMINI_API_KEY is not set; local metrics only",
  "cleanup_warnings": [],
  "upload_gain_db": 0.0
}
```

This excerpt omits the returned local `metrics`. Provider errors and invalid
model responses similarly preserve metrics, with a sanitized failure reason.
**Confidence is model self-assessment, not calibrated accuracy.** Live listening
quality requires evaluation against recordings with known changes.

### Comparison

B is RMS-matched to A **before** spectral comparison and listening. The response
retains original `metrics_a`/`metrics_b` and reports `matched_metrics_b`,
`loudness_match`, `deltas_b_minus_a` and `eq_direction_hints`. Linear gain alone
should not produce a tonal difference:

```json
{
  "loudness_match": {
    "applied": true,
    "gain_db_applied_to_b": 6.0206,
    "reason": null
  },
  "deltas_b_minus_a": {
    "relative_band_db": {"800_2000": 0.0},
    "centroid_hz": 0.0,
    "rolloff_85_hz": 0.0,
    "crest_factor_db": 0.0,
    "stereo": null
  }
}
```

This excerpt describes B at half A's amplitude. Deltas use **B minus A**;
positive band deltas mean relatively more energy in B. Direction hints suggest
cutting or boosting B toward A, not exact gain/EQ knobs. Silence makes RMS
matching undefined and is explicitly reported. Stereo deltas require two stereo
clips. Gemini compares both uploads in one request and returns differences,
which clip is brighter/more saturated/more compressed/more ambient, suggested
adjustments and uncertainties.

## Privacy and limitations

With a key configured, `analyze_audio` uploads the selected clip to Google,
and `compare_audio` uploads both selected clips. Only use recordings you have
permission to share. `audio_metrics` is always local. Temporary WAVs are deleted
on exit; uploaded files are deleted after the request, including most failure
paths. Cleanup failures are returned in `cleanup_warnings`. Remote handling and
retention are governed by Google's terms; deletion is not a zero-retention guarantee.
Local paths are included in local result metadata but not intentionally sent in
provider prompts. A user-supplied question/focus is sent to Google.

Uploads are resampled to **16 kHz mono** (sample rate, not 16 kbps bitrate).
Stereo and fine high-frequency detail must come from local metrics, not listening
claims. Severe stereo downmix cancellation (more than 40 dB power loss) disables
listening with an explicit reason; provide a mono guitar stem instead.
If loudness matching exceeds full scale, the same attenuation is applied to both
uploads and reported as `upload_gain_db`; original local measurements are unchanged.

Full mixes can confuse cymbals with guitar fizz, doubling with delay and other
instruments with guitar ambience/compression. Isolate the guitar where possible.
The tool cannot identify exact amps, pedals, cabinets or knob settings. Spectral
differences may reflect notes, pickup position, playing or arrangement rather
than EQ. Gemini's descriptions may be wrong; use evidence and listening checks.

## Recording and validation

Use the **same DI and phrase** for A/B, avoid clipping, compare at matched
loudness and choose short, focused clips. Record dry and processed versions when
possible; start/end should cover equivalent musical material.

```sh
.venv/bin/ruff check .
.venv/bin/pytest -q
.venv/bin/python scripts/make_validation_set.py --output /tmp/tone-validation
```

See [controlled validation](docs/validation.md) for known gain, soft clipping,
EQ and simple reverb variants. Tests are synthetic/offline with mocked Gemini;
the stdio integration test spawns the real server and exercises all four tools.
GitHub Actions runs lint and tests on Python 3.11 and 3.12 on push or manual
dispatch. Live Gemini checks are opt-in, require a key and incur API usage.

## License

MIT. No reference recordings or copyrighted song audio are included.