cochlea

A headless audio engine for agents. Write a score as data, render it offline to deterministic PCM, then listen through numbers — loudness, onsets, pitch, key, spectrograms — and assert what you heard. Compose → render → probe → verify, with no human ear (and no audio device) in the loop.

What the agent sees: the mel spectrogram of examples/scores/first_light.ron — the score used in the example below — after render and probe. No PCM in sight.

use cochlea_score::*;

let score = Score::new(SampleRate(48_000), Ppq(960))
    .time_signature(4, 4)
    .tempo(Ticks(0), Bpm(120.0))
    .track("lead", Instrument::preset("saw_lead"))
    .note("lead", bar(1).beat(1), Dur::quarter(), Pitch::A4, Vel(96))
    .automate("lead", Param::CUTOFF_HZ,
        keys![(bar(1), 400.0, ease_in_out()), (bar(3), 4_000.0)]);

let rendered = cochlea_render::render(&score)?;
rendered.write_wav("mix.wav")?;

use cochlea_verify::{VerifyExt, Tol, Ms, Cents, Db};
let report = rendered.verify(&score)
    .true_peak_below(-1.0)
    .pitch_matches_score("lead", Cents(10.0))
    .monotone("lead", Param::CUTOFF_HZ, bar(1)..bar(3))
    .silent_after(bar(5))
    .run();
assert!(report.passed);

Or entirely from the command line, score as RON:

cochlea render score.ron --out mix.wav --stems stems/ --verify
cochlea probe input.wav --json report.json --spectro spec.png
cochlea probe input.wav --digest --window-ms 500
cochlea diff a.wav b.wav --tier2
cochlea lint score.ron
cochlea spectro input.wav --out spec.png --sheet --bars-per-tile 8

cochlea probe works on any WAV — and FLAC, decoded bit-exact, still ffmpeg-free — no score required. That's the front door: point it at audio you didn't render and get the same JSON report and spectrogram an agent uses to review its own work.

How an agent listens

compose → render → probe (JSON) → spectrogram (one vision call) → verify

1. compose a score as data (RON, or the Rust builder above).

2. render it to deterministic PCM — cochlea render score.ron --out mix.wav.

3. probe the mix into a compact JSON report (loudness, onsets, pitch, key, silence, clipping) — cochlea probe mix.wav --json report.json. No image, no audio: the agent reads numbers.

4. look, when numbers aren't enough — cochlea spectro mix.wav --out spec.png renders one small PNG the agent reviews in a single vision call instead of reasoning about raw samples.

5. verify — cochlea render score.ron --verify runs the score's embedded assertions and exits nonzero on failure, so an agent can retry without a human confirming "yes, that sounds right."

The economics are the point, not an afterthought. The first_light render above is 7 seconds of 48 kHz/32-bit-float PCM and weighs 2.7 MB; a 3-minute piece at the same settings is ~66 MB — not something to hand an agent as text, let alone read sample-by-sample. Its probe report is ~2.5 KB of JSON (schema v2, trimmed here to the interesting fields — tempo/stereo/structure are new; this piece is short and close to mono, so they read near their floor here, see the digest below for a piece where they light up):

{
  "schema_version": 2,
  "source": { "sample_rate": 48000, "channels": 2, "duration_ms": 7035.708333333333 },
  "loudness": { "integrated_lufs": -22.70045487928478, "true_peak_dbtp": -15.910817022082783, "lra": 10.607660373688798 },
  "onsets": { "count": 6, "times_ms": [1077.33, 2149.33, 2346.67, 3221.33, 4538.67, 5034.67] },
  "pitch": { "voiced_ratio": 0.9847560975609756, "median_f0_hz": 110.00194603797897 },
  "key": { "tonic": "E", "mode": "major", "confidence": 0.8093960265638273 },
  "tempo": { "bpm": 55.97014925373134, "confidence": 0.003937017247067399, "clear_rhythm": false },
  "stereo": { "width": 0.02967719705208343, "correlation": 0.9981362354107913, "balance": -0.0016380539212361243 },
  "structure": { "section_count": 1, "confidence": 0.0 },
  "silence": { "trailing_ms": 2485.708333333333 },
  "clipping": { "clipped_samples": 0, "true_peak_over_0dbtp": false }
}

And the spectrogram is one small image. Here's the title_cue demo — a pad whose cutoff_hz automation sweeps 250 Hz → 5000 Hz across bars 1–3:

The dark band at the top of the frame narrows as the sweep runs — more high-frequency energy gets let through over time. An agent reads that directly off the image; the demo's Monotone(track: "pad", param: "cutoff_hz", ...) assertion checks the same thing numerically.

For a whole piece in one image regardless of length, --sheet tiles the spectrogram into a contact sheet instead of one long strip (two bars per tile here, --bars-per-tile 2):

Related MCP server: Audacity MCP Server

Reading audio without a context window

probe --digest skips JSON entirely and prints a deterministic text summary — one line per feature dimension, then a windowed timeline capped at ~40 rows. Real output for the drum_groove demo (20.8 s, four tracks, the wave-2 rhythm/stereo/structure dimensions in one screenful):

cochlea digest: 20.755s  2ch  48000Hz
loudness: integrated=-19.91  momentary_max=-17.54  true_peak=-4.78  lra=1.73
key: G major (conf 0.28)  pitch: voiced=26%  median=55.0Hz (A1 +0.5c)
tempo: 110.3bpm (conf 0.01) clear_rhythm=false
stereo: width=0.01 corr=1.00 bal=-0.00
structure: 1 section
onsets: count=32  rate=1.54/s
silence: leading=0ms  trailing=2545ms
clipping: clipped=0  over_0dbtp=false
timeline: window=1000ms  bucket=1x  rows=21
   idx        t(s)     rms   peak  ons     f0  flags
     0   0.000-1.000   -25.71  -7.38    2    55.0  -
     1   1.000-2.000   -25.60  -7.87    2    55.0  -
     ...
    17  17.000-18.000  -34.66  -16.80    1    54.7  -
    18  18.000-19.000  -58.77  -41.61    0       -  -
    19  19.000-20.000  -118.98 -103.16    0    54.7  S
    20  20.000-20.755  -161.14 -144.94    0    54.8  S

The tempo reads 110.3 BPM — matching the score's authored 110 BPM almost exactly — but clear_rhythm is false: a groove layering three simultaneous periodicities (sixteenth-note hats, quarter-note kick/snare, bar-level pad) dilutes autocorrelation confidence to 0.01, well under the 0.05 threshold calibrated on single-instrument click tracks (which measure 0.11–0.15). That's an honest reading, not a bug — demos/drum_groove asserts HasClearRhythm(expected: false) outright rather than tuning the detector to this one fixture.

cochlea diff compares two files in feature space instead of byte-for-byte — "did my change do what I meant," not "is the file bitwise equal." Real output diffing first_light.wav against title_cue.wav:

verdict: different (duration, loudness, onsets, key)
duration     a->b +1264.3 ms
loudness     integrated -6.22 LU  true_peak +6.02 dB  lra -8.91 LU
onsets       matched=0  mean_offset=-  max_offset=-  unmatched_a=6  unmatched_b=4
pitch        delta +1.0 cents
key          a=E major (conf 0.81)  b=A minor (conf 0.86)  changed=true
segments     max_abs_rms_delta 121.71 dB at idx=7
tempo        bpm +38.57 bpm  clear_rhythm_changed=false
stereo       width +0.00  correlation -0.00  balance +0.00
structure    section_count +0

Diff a render against itself, or a re-render of the same score, and the verdict reads byte-identical instead — the determinism contract above, checked from the outside. --tier2 turns that verdict into a gate: exit 0 for byte-identical or Tier-2-equivalent, exit 1 otherwise, so a CI job or an agent can catch a regression without ever reading a raw sample.

Agents as MCP clients

cochlea-mcp is a stdio MCP server over the same libraries the CLI uses — six tools (render_score, probe_audio, spectrogram, lint_score, probe_digest, audio_diff), each a thin wrapper over the matching library call, so any MCP client gets the same render → probe → spectrogram → verify loop as tool calls instead of shelled-out subprocesses:

cargo install --path crates/mcp
claude mcp add cochlea -- cochlea-mcp

Full tool schemas, arguments, and the JSON-RPC framing are in docs/mcp.md.

Install

Not on crates.io yet — build the cochlea binary from source:

git clone https://github.com/richer-richard/cochlea
cd cochlea
cargo install --path crates/cli

Concepts

• Score IR (cochlea-score): tracks, notes, per-parameter automation, a tempo map of step changes — all data, serializable as RON (version: 1, round-trip tested both ways). Positions are bar(3).beat(2), durations are exact fractions (Dur::quarter(), "3/16", dotted/triplet sugar); anything off the tick grid is an error, never a rounding.

• Integer time is ground truth. Ticks at 960 PPQ. BPM converts once to integer nanoseconds-per-quarter; tick→sample is exact rational u64/u128 arithmetic (via fenestra-anim's mul_div) applied once at event-schedule time. No accumulated floating-point seconds, no wall clock, property-tested drift-free over 10⁹ ticks.

• Synth (cochlea-synth): six presets over fundsp — sine, saw_lead, square_bass, chord_pad, noise_hat, pluck — plus a reverb insert. Instruments declare typed automatable params (name, unit, range, default); scores are validated against that registry. All noise is a counter-based RNG keyed (seed, sample_index) — random access, no stateful generator anywhere.

• Renderer (cochlea-render): 64-sample blocks split at event boundaries (note timing is sample-accurate; automation is control-rate, ~1.3 ms at 48 kHz). Tracks render independently — that's the parallelism unit and free stems. Voice allocation and oldest-note stealing are pure functions of the schedule. The master bus sums stems at f64 in fixed track order; the mix is byte-equal to the sum of the stems, by definition and by test.

• Features (cochlea-features): one schema-versioned JSON report — integrated LUFS / momentary max / true peak / LRA (via ebur128), spectral-flux onsets, YIN pitch with cents deviation, chroma + Krumhansl-Schmuckler key, tempo/beat tracking with a calibrated clear_rhythm flag, stereo width/correlation/balance, Foote novelty structure boundaries, silence/tail, clipping — plus a windowed segment timeline, an LLM-sized text digest, and a feature-space diff between two files.

• Spectro (cochlea-spectro): mel spectrogram PNGs (HTK filterbank, viridis, time ruler, bar markers) and tiled contact sheets so an agent reviews a whole piece in one vision call.

• Verify (cochlea-verify): the assertion DSL above, also embeddable in score RON under verify: — cochlea render score.ron --verify runs them and exits nonzero with a machine-readable JSON failure report.

Determinism, honestly stated

Audio is a fold, not a map: filters and delays carry state, so per-sample purity is not the contract. The contract is three tiers:

What buys Tier 1: the libm crate exclusively for transcendentals in DSP paths (std float methods are banned by clippy config, not convention), no fast-math, no implicit FMA (mul_add is banned too), denormals honored everywhere (flushing is a realtime hack and can't even be done uniformly across architectures — see docs/determinism.md), fixed summation order, f64 master bus, voices ticked sample-by-sample (fundsp's SIMD block path provably diverges from its scalar path and is banned), analysis FFTs on FftPlannerScalar (no runtime CPU dispatch). The full audit trail — per fundsp node family, ebur128 internals, rustfft dispatch — lives in docs/determinism.md.

Feature accuracy (synthesized ground truth, 48 kHz)

ffmpeg-free by design

cochlea reads WAV and FLAC (hound and symphonia, both pure Rust — FLAC is lossless, so decoded PCM is bit-exact by spec, checked against WAV twins in-tree), writes plain WAV, and renders PNGs on the CPU (rustfft + hand-rolled mel filterbank + viridis LUT + image). No subprocess calls, no system codecs, no GPU, no audio device — the entire pipeline is a pure Rust dependency graph, and CI bans GUI/GPU/device crates from ever entering Cargo.lock (deny.toml). Lossy formats (mp3/ogg) are the next compressed-format target; still ffmpeg-free.

Assertion cookbook

use cochlea_verify::{VerifyExt, Tol, Ms, Cents, Db};

rendered.verify(&score)
    // Mix-level loudness and headroom:
    .integrated_lufs(-14.0, Tol(0.5))     // streaming-loudness target
    .true_peak_below(-1.0)                 // intersample-safe headroom
    // Timing: did the hit land where the score says?
    .onset_at("drums", bar(17).beat(1), Ms(5.0))
    // Intonation: does every note read as written? (monophonic tracks)
    .pitch_matches_score("lead", Cents(10.0))
    // Did the filter sweep actually sweep? (authored curve, block-rate)
    .monotone("lead", Param::CUTOFF_HZ, bar(1)..bar(3))
    // Click detection away from note boundaries:
    .no_discontinuity("lead", Db(40.0))
    // Does the piece actually end?
    .silent_after(bar(64))
    .run();

The same assertions embed in score RON:

verify: [
    IntegratedLufs(target: -14.0, tol: 0.5),
    TruePeakBelow(dbtp: -1.0),
    OnsetAt(track: "drums", at: (17, 1), tol_ms: 5.0),
    PitchMatchesScore(track: "lead", tol_cents: 10.0),
    Monotone(track: "pad", param: "cutoff_hz", from: (1, 1), to: (3, 1), direction: Rising),
    NoDiscontinuity(track: "lead", db: 40.0),
    SilentAfter(at: (64, 1)),
]

cochlea render score.ron --verify runs them; failures come back as JSON ({"passed": false, "checks": [...]}) and a nonzero exit.

Four worked demos live in demos/: metronome (sample-exact scheduling, onset tolerances), chord_pad (harmony reads as written), title_cue (a four-bar cinematic sting asserting a LUFS target, a monotone filter sweep, click-freedom, and silence after the fade), and drum_groove (a 110 BPM, eight-bar drum groove asserting detected tempo, stereo width, loudness range, and section count — and an honest HasClearRhythm(false), since a layered real-instrument groove dilutes the tempo detector's confidence below its click-track-calibrated threshold even though the BPM itself lands almost exactly on target).

Workspace

crates/
  score      # IR: ticks, tempo map, bar/beat math, notes, automation, RON form
  synth      # Patch trait over fundsp, six presets, param registry, counter RNG
  render     # block engine, voices, stems, f64 master sum, WAV out
  features   # LUFS/true peak, onsets, pitch, chroma/key, tempo, stereo, structure, LRA
  decode     # WAV + FLAC -> Audio (hound + symphonia, both pure Rust)
  spectro    # mel spectrogram -> PNG, contact sheets, image diff
  verify     # assertion DSL + RON-embeddable specs + JSON reports
  cli        # the `cochlea` binary
  mcp        # MCP stdio server (agents call render/probe/verify as tools)

features and spectro depend on neither score nor synth — enforced in CI — which is why probe works on arbitrary WAVs (and, via decode, FLAC).

License

MIT OR Apache-2.0, at your option.