cochlea

A headless audio engine for agents. Write a score as data, render it offline to deterministic PCM, then listen through numbers — loudness, onsets, pitch, key, spectrograms — and assert what you heard. Compose → render → probe → verify, with no human ear (and no audio device) in the loop.

What the agent sees: the mel spectrogram of examples/scores/first_light.ron — the score used in the example below — after render and probe. No PCM in sight.

use cochlea_score::*;

let score = Score::new(SampleRate(48_000), Ppq(960))
    .time_signature(4, 4)
    .tempo(Ticks(0), Bpm(120.0))
    .track("lead", Instrument::preset("saw_lead"))
    .note("lead", bar(1).beat(1), Dur::quarter(), Pitch::A4, Vel(96))
    .automate("lead", Param::CUTOFF_HZ,
        keys![(bar(1), 400.0, ease_in_out()), (bar(3), 4_000.0)]);

let rendered = cochlea_render::render(&score)?;
rendered.write_wav("mix.wav")?;

use cochlea_verify::{VerifyExt, Tol, Ms, Cents, Db};
let report = rendered.verify(&score)
    .true_peak_below(-1.0)
    .pitch_matches_score("lead", Cents(10.0))
    .monotone("lead", Param::CUTOFF_HZ, bar(1)..bar(3))
    .silent_after(bar(5))
    .run();
assert!(report.passed);

Or entirely from the command line, score as RON:

cochlea render score.ron --out mix.wav --stems stems/ --verify
cochlea probe input.wav --json report.json --spectro spec.png
cochlea probe input.wav --digest --window-ms 500
cochlea probe input.mp3 --from 42.0 --to 44.5      # zoom into a window, any format
cochlea diff a.wav b.wav --tier2 --spectro delta.png
cochlea lint score.ron
cochlea spectro input.wav --out spec.png --annotate  # draw beats/onsets/pitch on the image
cochlea import song.mid --out score.ron              # SMF -> score, timing exact
cochlea reference    # the full score-authoring reference, generated from the live preset bank

cochlea probe works on any WAV — and FLAC (decoded bit-exact), mp3, and ogg, still ffmpeg-free — no score required. That's the front door: point it at audio you didn't render and get the same JSON report and spectrogram an agent uses to review its own work.

How an agent listens

compose → render → probe (JSON) → spectrogram (one vision call) → verify

1. compose a score as data (RON, or the Rust builder above).

2. render it to deterministic PCM — cochlea render score.ron --out mix.wav.

3. probe the mix into a compact JSON report (loudness, onsets, pitch, key, silence, clipping) — cochlea probe mix.wav --json report.json. No image, no audio: the agent reads numbers.

4. look, when numbers aren't enough — cochlea spectro mix.wav --out spec.png renders one small PNG the agent reviews in a single vision call instead of reasoning about raw samples.

5. verify — cochlea render score.ron --verify runs the score's embedded assertions and exits nonzero on failure, so an agent can retry without a human confirming "yes, that sounds right."

And when something in the middle of a long file needs a closer listen, every read tool takes --from/--to: probe just bars 17–19, spectrogram just the drop. The cut is frame-exact, report times are relative to it, and source.start_ms says where it came from — the tier stack becomes a zoom lens instead of a whole-file-only report.

The economics are the point, not an afterthought. The first_light render above is 7 seconds of 48 kHz/32-bit-float PCM and weighs 2.7 MB; a 3-minute piece at the same settings is ~66 MB — not something to hand an agent as text, let alone read sample-by-sample. Its probe report is a few KB of JSON (schema v4, trimmed here to the interesting fields — note pitch.melody: the piece's bass line and melody read back as note events, the compose loop's read-back half):

{
  "schema_version": 4,
  "source": { "sample_rate": 48000, "channels": 2, "duration_ms": 7035.708333333333, "start_ms": 0.0 },
  "loudness": { "integrated_lufs": -22.700454879284784, "true_peak_dbtp": -15.910817022082783, "lra": 10.607660373688798 },
  "onsets": { "count": 6, "times_ms": [1077.33, 2149.33, 2346.67, 3221.33, 4538.67, 5034.67] },
  "pitch": { "voiced_ratio": 0.9847560975609756, "median_f0_hz": 110.00194603797897,
             "melody": [ { "name": "A2", "start_ms": 0.0, "end_ms": 1045.3, "cents_off": 0.1 },
                         { "name": "E2", "start_ms": 1077.3, "end_ms": 2112.0, "cents_off": 0.3 },
                         { "name": "F#2", "start_ms": 2154.7, "end_ms": 3178.7, "cents_off": 0.2 },
                         { "name": "E2", "start_ms": 3210.7, "end_ms": 4384.0, "cents_off": 0.3 },
                         { "name": "E5", "start_ms": 4394.7, "end_ms": 5813.3, "cents_off": -0.4 } ] },
  "timbre": { "mfcc_mean": [-37.64, 14.47, -4.44, 1.24, "..."], "mfcc_std": ["..."], "frames": 656 },
  "key": { "tonic": "E", "mode": "major", "confidence": 0.8093960265638273 },
  "tempo": { "bpm": 55.97014925373134, "confidence": 0.6633739386089712, "stability": 0.3333333333333333,
             "candidates": [ { "bpm": 55.97014925373134, "salience": 0.6633739386089712 },
                             { "bpm": 112.5, "salience": 0.21588204941945222 } ] },
  "rhythm": { "grid_alignment": 0.8333333333333334, "grid": "straight", "offbeat_ratio": 0.4, "clear_rhythm": true },
  "stereo": { "width": 0.02967719705208343, "correlation": 0.9981362354107913, "balance": -0.0016380539212361243 },
  "structure": { "section_count": 1, "confidence": 0.0 },
  "silence": { "trailing_ms": 2485.708333333333 },
  "clipping": { "clipped_samples": 0, "true_peak_over_0dbtp": false }
}

And the spectrogram is one small image. Here's the title_cue demo — a pad whose cutoff_hz automation sweeps 250 Hz → 5000 Hz across bars 1–3:

The dark band at the top of the frame narrows as the sweep runs — more high-frequency energy gets let through over time. An agent reads that directly off the image; the demo's Monotone(track: "pad", param: "cutoff_hz", ...) assertion checks the same thing numerically.

For a whole piece in one image regardless of length, --sheet tiles the spectrogram into a contact sheet instead of one long strip (two bars per tile here, --bars-per-tile 2):

Related MCP server: Talky Talky

Reading audio without a context window

probe --digest skips JSON entirely and prints a deterministic text summary — one line per feature dimension, then a windowed timeline capped at ~40 rows. Real output for the drum_groove demo (20.8 s, four tracks, the wave-2 rhythm/stereo/structure dimensions in one screenful):

cochlea digest: 20.755s  2ch  48000Hz
loudness: integrated=-24.06  momentary_max=-22.42  true_peak=-5.95  lra=1.61
key: A# minor (conf 0.54)  pitch: voiced=23%  median=63.8Hz (C2 -42.8c)
melody: 6 notes  C2 C2 C2 C2 A1 A1
tempo: 110.3bpm (conf 0.79, stability 1.00)  alts: 54.9bpm(0.89), 36.6bpm(0.79)
rhythm: clear=true  grid_align=0.98 (straight)  offbeat=0.56
stereo: width=0.07 corr=0.99 bal=-0.01
structure: 1 section
onsets: count=58  rate=2.79/s
silence: leading=0ms  trailing=2545ms
clipping: clipped=0  over_0dbtp=false
timeline: window=1000ms  bucket=1x  rows=21
   idx        t(s)     rms   peak  ons     f0  flags
     0   0.000-1.000   -25.55  -7.36    4    64.0  -
     1   1.000-2.000   -25.61  -8.37    3    63.4  -
     ...

Tempo and rhythm are reported as separate axes, because they change independently — a drum solo can hold a rock-steady pulse while its pattern turns unrecognizable, and that difference is exactly what an agent needs to see. Here the tempo reads 110.3 BPM (matching the authored 110), stability 1.00 says the speed never moves across the piece, and the alts list surfaces the genuine half-tempo reading (54.9 BPM, salience 0.89 — actually the stronger raw peak; the octave prior breaks the tie toward the beat). Metrical ambiguity is data an agent can weigh, not a coin flip hidden inside the detector. The rhythm line then reports how the hits relate to that pulse: 98% of onsets sit on the beat-subdivision grid, 56% of them on off-beat subdivisions (an eighth-note hat groove — syncopation as a number), so clear=true. (Under the pre-0.2.0 metric this same groove read clear_rhythm=false at confidence 0.01 — layering hats, kick, snare, and pad across three metrical levels diluted every lag's share of a mass-fraction score. The grid-based rule asks the right question instead.)

The (straight) tag is the grid hypothesis test: alignment is measured against both straight sixteenths and eighth-note triplets, and the report carries whichever more hits land on. A shuffle or swing take reads grid_align=1.00 (triplet) — recognized as an aligned triplet rhythm — instead of being force-fit to sixteenths and scored sloppy.

cochlea diff compares two files in feature space instead of byte-for-byte — "did my change do what I meant," not "is the file bitwise equal." Real output diffing first_light.wav against title_cue.wav:

verdict: different (duration, loudness, onsets, key)
duration     a->b +1264.3 ms
loudness     integrated -5.95 LU  true_peak +5.70 dB  lra -8.88 LU
onsets       matched=0  mean_offset=-  max_offset=-  unmatched_a=6  unmatched_b=5
pitch        delta +0.5 cents
key          a=E major (conf 0.81)  b=A minor (conf 0.86)  changed=true
segments     max_abs_rms_delta 120.99 dB at idx=7
tempo        bpm -24.01 bpm  stability -0.33
rhythm       clear_rhythm_changed=false  grid_align -0.03  grid_changed=true
timbre       mfcc_distance 4.00
stereo       width +0.14  correlation -0.08  balance -0.01
structure    section_count +0

The timbre row is an MFCC spectral-shape distance (c0, which is just loudness, excluded): the same instrument re-rendered measures ~0, a sine-for-saw swap at matched loudness measures well above it — the "did the re-render keep the instrument's character" axis that loudness and pitch can't see. Add --spectro delta.png and the diff also renders a signed difference heat map: red where B is louder, blue where it's quieter, black where nothing changed — a moved onset is a blue/red vertical pair, a brightened sweep a red wedge. What changed becomes visible structure, not just a number.

Diff a render against itself, or a re-render of the same score, and the verdict reads byte-identical instead — the determinism contract above, checked from the outside. --tier2 turns that verdict into a gate: exit 0 for byte-identical or Tier-2-equivalent, exit 1 otherwise, so a CI job or an agent can catch a regression without ever reading a raw sample.

Agents as MCP clients

cochlea-mcp is a stdio MCP server over the same libraries the CLI uses — eight tools (render_score, probe_audio, spectrogram, lint_score, probe_digest, audio_diff, import_midi, score_reference), each a thin wrapper over the matching library call, so any MCP client gets the same compose → render → probe → spectrogram → verify loop as tool calls instead of shelled-out subprocesses:

cargo install cochlea-mcp
claude mcp add cochlea -- cochlea-mcp

What makes it agent-native rather than a CLI in a trenchcoat:

• It teaches itself. score_reference returns the complete authoring reference — the RON grammar, the live preset catalog with every automatable parameter (generated from the same registry that validates scores, so it can't go stale), all verify: assertions, and a worked example the test suite itself renders. An agent connected cold can compose without ever seeing this repo.

• It shows, not points. spectrogram returns the image inline as MCP image content (base64 PNG, size-capped), so a client with no filesystem access still gets the one-vision-call review; out_path is optional. annotate: true draws the detected beats, onsets, and pitch onto the image, and audio_diff can return the signed difference heat map the same way.

• It zooms. probe_audio and spectrogram take from_s/to_s — lean into 42.0–44.5 s of a long file the way a human replays a bar, instead of paying for whole-file analysis every call.

• It can be confined. cochlea-mcp --root DIR refuses any read or write that resolves (canonically — symlinks and .. included) outside DIR, before touching the filesystem.

Full tool schemas, arguments, and the JSON-RPC framing are in docs/mcp.md.

Install

All nine crates are on crates.io:

cargo install cochlea        # the CLI: render / probe / diff / lint / spectro / reference
cargo install cochlea-mcp    # the MCP stdio server
cargo add cochlea-features   # or any crate as a library dependency

Or from source: git clone https://github.com/richer-richard/cochlea && cd cochlea && cargo install --path crates/cli.

Concepts

• Score IR (cochlea-score): tracks, notes, per-parameter automation, a tempo map of step changes, an optional master section — all data, serializable as RON (version: 1, round-trip tested both ways). Positions are bar(3).beat(2), durations are exact fractions (Dur::quarter(), "3/16", dotted/triplet sugar); anything off the tick grid is an error, never a rounding. Standard MIDI Files import with timing intact (cochlea import — SMF ticks land on the grid verbatim, GM programs map to labeled preset guesses).

• Integer time is ground truth. Ticks at 960 PPQ. BPM converts once to integer nanoseconds-per-quarter; tick→sample is exact rational u64/u128 arithmetic (via fenestra-anim's mul_div) applied once at event-schedule time. No accumulated floating-point seconds, no wall clock, property-tested drift-free over 10⁹ ticks.

• Synth (cochlea-synth): eight presets over fundsp — sine, saw_lead, square_bass, chord_pad (genuinely stereo: its detuned saws pan apart), noise_hat, pluck, kick, snare — plus a reverb insert. Instruments declare typed automatable params (name, unit, range, default); scores are validated against that registry, and the same registry generates the self-describing authoring reference. All noise is a counter-based RNG keyed (seed, sample_index) — random access, no stateful generator anywhere.

• Renderer (cochlea-render): 64-sample blocks split at event boundaries (note timing is sample-accurate; automation is control-rate, ~1.3 ms at 48 kHz). Tracks render independently — that's the parallelism unit and free stems. Voice allocation and oldest-note stealing are pure functions of the schedule. The master bus sums stems at f64 in fixed track order, then runs the score's optional master stage: an output gain and a brick-wall lookahead limiter whose sample-peak ceiling holds exactly (offline lookahead is a forward window maximum, no delay line) — the tool for hitting LUFS targets with TruePeakBelow headroom. Without a master section, the mix is byte-equal to the sum of the stems, by definition and by test.

• Features (cochlea-features): one schema-versioned JSON report — integrated LUFS / momentary max / true peak / LRA (via ebur128), spectral-flux onsets, YIN pitch with cents deviation plus a quantized melody (note events an agent can diff against what it wrote), an MFCC timbre digest, chroma + Krumhansl-Schmuckler key, tempo (pulse clarity, octave-alternative candidates, windowed stability) and rhythm (grid alignment with a straight-vs-triplet hypothesis test, offbeat ratio, a calibrated clear_rhythm) as separate axes, stereo width/correlation/balance, Foote novelty structure boundaries, silence/tail, clipping — plus a windowed segment timeline, an LLM-sized text digest, a feature-space diff between two files, and frame-exact windowing (Audio::window) behind every --from/--to.

• Spectro (cochlea-spectro): mel spectrogram PNGs (HTK filterbank, viridis, time ruler, bar markers), analysis overlays (beat grid, onsets, pitch drawn on the image), signed A→B difference heat maps, and tiled contact sheets so an agent reviews a whole piece in one vision call.

• Verify (cochlea-verify): the assertion DSL above, also embeddable in score RON under verify: — cochlea render score.ron --verify runs them and exits nonzero with a machine-readable JSON failure report.

Determinism, precisely scoped

Audio is a fold, not a map: filters and delays carry state, so per-sample purity is not the contract. The contract is three tiers:

What buys Tier 1: the libm crate exclusively for transcendentals in DSP paths (std float methods are banned by clippy config, not convention), no fast-math, no implicit FMA (mul_add is banned too), denormals honored everywhere (flushing is a realtime hack and can't even be done uniformly across architectures — see docs/determinism.md), fixed summation order, f64 master bus, voices ticked sample-by-sample (fundsp's SIMD block path provably diverges from its scalar path and is banned), analysis FFTs on FftPlannerScalar (no runtime CPU dispatch). The full audit trail — per fundsp node family, ebur128 internals, rustfft dispatch — lives in docs/determinism.md.

Feature accuracy (synthesized ground truth, 48 kHz)

ffmpeg-free by design

cochlea reads WAV, FLAC, mp3, and ogg/vorbis (hound and symphonia, both pure Rust), writes plain WAV, and renders PNGs on the CPU (rustfft

• hand-rolled mel filterbank + viridis LUT + image). No subprocess calls, no system codecs, no GPU, no audio device — the entire pipeline is a pure Rust dependency graph, and CI bans GUI/GPU/device crates from ever entering Cargo.lock (deny.toml). Two decode contracts, stated where they live: WAV/FLAC decode bit-exact (FLAC is lossless by spec, checked against WAV twins in-tree); mp3/ogg are analysis input only — reproducible per build, but a codec already threw the original samples away, so no exactness claim exists to make.

Assertion cookbook

use cochlea_verify::{VerifyExt, Tol, Ms, Cents, Db};

rendered.verify(&score)
    // Mix-level loudness and headroom:
    .integrated_lufs(-14.0, Tol(0.5))     // streaming-loudness target
    .true_peak_below(-1.0)                 // intersample-safe headroom
    // Timing: did the hit land where the score says?
    .onset_at("drums", bar(17).beat(1), Ms(5.0))
    // Intonation: does every note read as written? (monophonic tracks)
    .pitch_matches_score("lead", Cents(10.0))
    // Was the sweep *written*? (authored curve, block-rate — a score lint)
    .monotone("lead", Param::CUTOFF_HZ, bar(1)..bar(3))
    // ...and did it audibly *happen*? (rendered stem's spectral centroid)
    .brightness_rises("lead", bar(1)..bar(3), 1.3)
    // Do the hits land on the detected beat grid?
    .grid_alignment_at_least(0.9)
    // Click detection away from note boundaries:
    .no_discontinuity("lead", Db(40.0))
    // Does the piece actually end?
    .silent_after(bar(64))
    .run();

The same assertions embed in score RON:

verify: [
    IntegratedLufs(target: -14.0, tol: 0.5),
    TruePeakBelow(dbtp: -1.0),
    OnsetAt(track: "drums", at: (17, 1), tol_ms: 5.0),
    PitchMatchesScore(track: "lead", tol_cents: 10.0),
    Monotone(track: "pad", param: "cutoff_hz", from: (1, 1), to: (3, 1), direction: Rising),
    BrightnessRises(track: "pad", from: (1, 1), to: (3, 1), min_ratio: 1.3),
    NoDiscontinuity(track: "lead", db: 40.0),
    SilentAfter(at: (64, 1)),
    TempoIs(bpm: 110.0, tol_bpm: 2.0),
    HasClearRhythm(expected: true),
    GridAlignmentAtLeast(min: 0.9),
]

cochlea render score.ron --verify runs them; failures come back as JSON ({"passed": false, "checks": [...]}) and a nonzero exit.

To actually hit a loudness target rather than just assert it, give the score a master bus — gain to push, a limiter to hold the ceiling:

master: Master(
    gain_db: 4.0,
    limiter: Limiter(ceiling_db: -2.0),   // sample-peak ceiling holds exactly
),
verify: [
    IntegratedLufs(target: -14.0, tol: 0.5),
    TruePeakBelow(dbtp: -1.0),   // ~1 dB headroom over the ceiling: true peak is inter-sample
]

Four worked demos live in demos/: metronome (sample-exact scheduling, onset tolerances), chord_pad (harmony reads as written), title_cue (a four-bar cinematic sting asserting a LUFS target, a monotone filter sweep, click-freedom, and silence after the fade), and drum_groove (a 110 BPM eight-bar groove on the real kick and snare patches, hats panned right and snare left, asserting detected tempo, HasClearRhythm(true) with grid alignment ≥ 0.9, stereo width, loudness range, and section count — the fixture that motivated the tempo/rhythm split, since the old confidence metric read it as rhythm-less at 0.01 despite a spot-on BPM).

Workspace

crates/
  score      # IR: ticks, tempo map, bar/beat math, notes, automation, master, RON form, MIDI import
  synth      # Patch trait over fundsp, eight presets, param registry, counter RNG
  render     # block engine, voices, stems, f64 master sum + gain/limiter, WAV out
  features   # LUFS/true peak, onsets, pitch+melody, timbre, chroma/key, tempo, rhythm, stereo, structure
  decode     # WAV + FLAC (bit-exact) + mp3 + ogg (analysis) -> Audio, pure Rust
  spectro    # mel spectrogram -> PNG, overlays, diff heat maps, contact sheets
  verify     # assertion DSL + RON-embeddable specs + JSON reports
  cli        # the `cochlea` binary
  mcp        # MCP stdio server (agents call compose/render/probe/verify as tools)

features and spectro depend on neither score nor synth — enforced in CI — which is why probe works on arbitrary audio files with no score in sight.

License

MIT OR Apache-2.0, at your option.