# peacelingo-stt-bench

An MCP server that lets an AI agent compare speech-to-text providers on the same short clip
and get back error rates (CER, WER), latency and estimated cost. It was built for the
PeaceLingo project and works on its own. Providers: Soniox, Groq Whisper, AssemblyAI.
Transport: stdio only. Built on the official Python SDK (`mcp`, FastMCP).

> Transcripts are untrusted data from third-party services. The server returns them as
> JSON text only, cut to 2,000 characters, and never executes them or builds commands from
> them. Agents and people reading results should treat them the same way.

## Architecture

```mermaid
flowchart LR
  U["Person: which STT is best for this clip?"] --> LLM
  subgraph HOST["1. Host app (for example Claude Code)"]
    LLM["AI agent picks a tool"] --> CLI["MCP client (built into the host)"]
  end
  CLI -->|"JSON-RPC 2.0 over stdio (your machine)"| SRV
  subgraph SERVER["2. peacelingo-stt-bench (this repo, FastMCP)"]
    SRV["server.py"] --> T1["compare_stt: same clip to many providers"]
    SRV --> T2["list_providers: which are configured"]
    SRV --> T3["estimate_cost: hourly price math"]
  end
  T1 --> V1["Soniox"]
  T1 --> V2["Groq Whisper"]
  T1 --> V3["AssemblyAI"]
  V1 --> M
  V2 --> M
  V3 --> M
  M["3. Metrics: CER, WER, latency, cost"] --> CLI
```

Per call: the agent chooses `compare_stt`; the client sends it to the server over stdio;
the server validates the path, reserves its call budget, sends the clip to each provider that
has a key, scores each transcript against the reference sentence, and returns a JSON table.
MCP also defines Resources and Prompts; this server does not use them yet.

## Install

Requires [uv](https://docs.astral.sh/uv/) and Python 3.11+.

```sh
git clone <this repository> peacelingo-stt-bench
cd peacelingo-stt-bench
uv sync
uv run pytest -q          # tests, no network, no keys
```

Attach it to Claude Code:

```sh
claude mcp add peacelingo-stt-bench -- uv run --directory /absolute/path/to/peacelingo-stt-bench stt-bench
```

## Tools

| Tool | What it does |
| --- | --- |
| `compare_stt(audio_path, language?, reference?, providers?)` | Sends one clip to every provider that has a key (or the ones you name). Returns per provider: status, model, transcript (max 2,000 chars), CER, WER, latency in seconds, estimated cost in USD. `reference` defaults to `clips/<name>.txt`. |
| `list_providers()` | Providers, model, whether a key is configured (never the key), hourly price and its source, provider calls left today. |
| `estimate_cost(duration_seconds, providers?)` | Cost estimate from the built-in hourly price table in `src/stt_bench/cost.py`. A price that was not found on the provider page is `null`, never guessed. |

A provider without a key is reported as `no_key_configured` and costs no budget, so the tool
works with any subset of providers.

## Key setup

Each person issues their own keys and keeps them private. Never commit a key.

| Provider | Environment variable | Or a file (mode 600) |
| --- | --- | --- |
| Soniox | `SONIOX_API_KEY` | `~/.config/peacelingo/soniox.env` |
| Groq | `GROQ_API_KEY` | `~/.config/peacelingo/groq.env` |
| AssemblyAI | `ASSEMBLYAI_API_KEY` | `~/.config/peacelingo/assemblyai.env` |

A file holds one line such as `GROQ_API_KEY=...`. The environment variable wins if both exist.
Use a dedicated development key with a spending limit, not a production key.

## Clips

`clips/` holds three synthetic 16 kHz mono wav clips (English, Korean, Spanish) made with the
macOS `say` voice, plus the reference sentence in `<name>.txt`. Regenerate with
`make clips`. There are no real recordings. Audio is only accepted from this folder.

## Tests and checks

```sh
make test     # pytest
make eval     # agent-style call sequence with every provider faked
make scan     # pip-audit + gitleaks over the tree and the git history
make smoke    # opt-in: one real call per provider on the English clip (STT_BENCH_LIVE=1)
```

The smoke test uses your own keys and a few cents of provider credit at most. Install the
pre-commit hook with `pre-commit install` to run gitleaks before every commit.

## Security

Short version; the full threat model is in [SECURITY.md](SECURITY.md).

- `audio_path` must resolve inside `clips/`; no URLs, no `..`, no escaping symlinks, only
  wav/mp3/m4a, at most 10 MB and 30 seconds.
- Keys are read only from the environment or `~/.config/peacelingo/*.env`; never printed,
  logged or returned; error text is masked.
- At most 3 provider calls per tool call and 50 per day (`STT_BENCH_DAILY_CAP`), counted in a
  locked local file.
- Stdio only; logs go to stderr; stdout carries protocol messages only.
- Timeouts on every HTTP call, TLS verification on, redirects are never followed.
- Dependencies are pinned with hashes in `uv.lock`; `make audit` runs pip-audit.
- Audio you send goes to the providers you configured, under their terms. Soniox and
  AssemblyAI transcriptions are deleted from the provider right after the call.
- Remote access (Streamable HTTP, OAuth) is not implemented; see `docs/oauth-plan.md`
  (planned).

## Limits

- Prices are a snapshot dated in `cost.py`; check the provider pages before relying on them.
- mp3/m4a length is bounded by file size, not decoded.
- One synthetic clip per language is a smoke check, not a benchmark.

## License

MIT, see [LICENSE](LICENSE).