peacelingo-stt-bench

An MCP server that lets an AI agent compare speech-to-text providers on the same short clip and get back error rates (CER, WER), latency and estimated cost. It was built for the PeaceLingo project and works on its own. Providers: Soniox, Groq Whisper, AssemblyAI. Transport: stdio only. Built on the official Python SDK (mcp, FastMCP).

Transcripts are untrusted data from third-party services. The server returns them as JSON text only, cut to 2,000 characters, and never executes them or builds commands from them. Agents and people reading results should treat them the same way.

Architecture

flowchart LR
  U["Person: which STT is best for this clip?"] --> LLM
  subgraph HOST["1. Host app (for example Claude Code)"]
    LLM["AI agent picks a tool"] --> CLI["MCP client (built into the host)"]
  end
  CLI -->|"JSON-RPC 2.0 over stdio (your machine)"| SRV
  subgraph SERVER["2. peacelingo-stt-bench (this repo, FastMCP)"]
    SRV["server.py"] --> T1["compare_stt: same clip to many providers"]
    SRV --> T2["list_providers: which are configured"]
    SRV --> T3["estimate_cost: hourly price math"]
  end
  T1 --> V1["Soniox"]
  T1 --> V2["Groq Whisper"]
  T1 --> V3["AssemblyAI"]
  V1 --> M
  V2 --> M
  V3 --> M
  M["3. Metrics: CER, WER, latency, cost"] --> CLI

Per call: the agent chooses compare_stt; the client sends it to the server over stdio; the server validates the path, reserves its call budget, sends the clip to each provider that has a key, scores each transcript against the reference sentence, and returns a JSON table. MCP also defines Resources and Prompts; this server does not use them yet.

Related MCP server: AudioPod MCP Server

Install

Requires uv and Python 3.11+.

git clone <this repository> peacelingo-stt-bench
cd peacelingo-stt-bench
uv sync
uv run pytest -q          # tests, no network, no keys

Attach it to Claude Code:

claude mcp add peacelingo-stt-bench -- uv run --directory /absolute/path/to/peacelingo-stt-bench stt-bench

Tools

A provider without a key is reported as no_key_configured and costs no budget, so the tool works with any subset of providers.

Key setup

Each person issues their own keys and keeps them private. Never commit a key.

A file holds one line such as GROQ_API_KEY=.... The environment variable wins if both exist. Use a dedicated development key with a spending limit, not a production key.

Clips

clips/ holds three synthetic 16 kHz mono wav clips (English, Korean, Spanish) made with the macOS say voice, plus the reference sentence in <name>.txt. Regenerate with make clips. There are no real recordings. Audio is only accepted from this folder.

Tests and checks

make test     # pytest
make eval     # agent-style call sequence with every provider faked
make scan     # pip-audit + gitleaks over the tree and the git history
make smoke    # opt-in: one real call per provider on the English clip (STT_BENCH_LIVE=1)

The smoke test uses your own keys and a few cents of provider credit at most. Install the pre-commit hook with pre-commit install to run gitleaks before every commit.

Security

Short version; the full threat model is in SECURITY.md.

• audio_path must resolve inside clips/; no URLs, no .., no escaping symlinks, only wav/mp3/m4a, at most 10 MB and 30 seconds.

• Keys are read only from the environment or ~/.config/peacelingo/*.env; never printed, logged or returned; error text is masked.

• At most 3 provider calls per tool call and 50 per day (STT_BENCH_DAILY_CAP), counted in a locked local file.

• Stdio only; logs go to stderr; stdout carries protocol messages only.

• Timeouts on every HTTP call, TLS verification on, redirects are never followed.

• Dependencies are pinned with hashes in uv.lock; make audit runs pip-audit.

• Audio you send goes to the providers you configured, under their terms. Soniox and AssemblyAI transcriptions are deleted from the provider right after the call.

• Remote access (Streamable HTTP, OAuth) is not implemented; see docs/oauth-plan.md (planned).

Limits

• Prices are a snapshot dated in cost.py; check the provider pages before relying on them.

• mp3/m4a length is bounded by file size, not decoded.

• One synthetic clip per language is a smoke check, not a benchmark.

License

MIT, see LICENSE.