FireRedTTS3 — multilingual TTS (24 languages, incl. Ukrainian)

Local speech synthesis service based on FireRedTTS3 with an HTTP API and MCP server. Third in the row alongside ukrainian-tts (StyleTTS2, :8000) and HolosTTS (:8010); this one runs on :8020.

The model was released on 13.08.2026, licensed under Apache 2.0, weights are public.

How it differs from its neighbors

Three things worth understanding before you start:

1. There are no preset voices. An empty library means there is nothing to synthesize with. First you add a voice from any speech recording (add_firered_voice), then use it by name. The model clones zero-shot right during synthesis.

2. A reference transcript is required. The model needs more than just the audio — it also needs the text of what is spoken in it. If you don't provide it, we transcribe with Whisper, but your own text is always more accurate.

3. No stress marks. No dictionary, no ByT5 fallback, no manual Mu+droho, unlike the neighbors. The model places stress itself from context, and on homographs (zámok/zamók) it will confuse them. This is a fundamental limitation of a multilingual model — if stress marks are critical, HolosTTS is still the better choice for pure Ukrainian.

Related MCP server: STT2TTS MCP

Installation

cd /Users/admin/Projects/firered-tts
./setup.sh                    # venv + залежності + апстрім + патч + ваги

setup.sh does four things:

1. .venv with Python 3.11, torch 2.8.0 for MPS/CPU

2. clones upstream into vendor/FireRedTTS3 at pinned commit 00570ad

3. patches it for Apple Silicon — see below

4. pulls base+redae weights (~11.4 GB) into pretrained_models/

Weights separately (takes a while — better run detached):

nohup ./download-weights.sh base > data/download.log 2>&1 &
./download-weights.sh instruct   # +7.9 ГБ, для дизайну голосу й редагування

Why the patch

The upstream is written exclusively for NVIDIA and does not run on Mac at all:

flash_attn is CUDA-only and cannot be built on Metal at all; sdpa works everywhere, including on NVIDIA, so the patch breaks nothing.

The tenth spot in the patch stands apart: base.py:317 — this is not about portability but performance (reference encoding cache, see "Speed" below).

src/patch_upstream.py is idempotent and fails if the replacement does not land — if the upstream has changed, you will find out immediately, not via a CUDA error on the first synthesis.

Running

./run.sh                      # http://localhost:8020

Running it separately is not required — the MCP server will start the backend itself. The backend shuts down after 1800s of idle time (IDLE_SHUTDOWN_SECONDS) and frees memory.

run.sh holds a lock on the port (data/.start-<port>.lock): a second launch while the service is alive simply exits with code 0. This is intentional — several clients trigger autostart at once, and /health stays silent for all ~90s of loading.

HTTP API

# 1) завести голос
curl -X POST localhost:8020/clone_voice -H 'Content-Type: application/json' -d '{
  "audio": "prompts/зразок.wav", "name": "Богдан",
  "prompt_text": "Це зразок мого голосу для клонування.",
  "language": "Ukrainian", "gender": "male"}'

# 2) озвучити
curl -X POST localhost:8020/tts -H 'Content-Type: application/json' -d '{
  "text": "Сьогодні чудова погода, ходімо гуляти в парк.",
  "voice": "Богдан", "format": "mp3"}' -o out.mp3

# Або one-shot — без реєстрації голосу: передай reference_audio замість voice,
# транскрипт зробить Whisper (перший виклик +~30с, далі кешується)
curl -X POST localhost:8020/tts -H 'Content-Type: application/json' -d '{
  "text": "Сьогодні чудова погода.", "reference_audio": "prompts/зразок.mp3",
  "format": "mp3"}' -o out.mp3

MCP

See mcp_server/README.md. In short:

claude mcp add firered-tts -- /Users/admin/Projects/firered-tts/.venv/bin/python \
  /Users/admin/Projects/firered-tts/mcp_server/server.py

Tools: firered_backend_status, list_firered_voices, add_firered_voice, delete_firered_voice, synthesize_firered_speech, design_firered_voice, edit_firered_speech.

Memory and speed

Mac mini M4, 16 GB. On disk base (7.9) + redae (3.5) = 11.4 GB, but ~7.7 GB goes into memory: the LLM backbone loads in fp16 (4.2 instead of 8.4 — see optimization 2 below), the rest stays fp32.

Weights are not the whole picture. Measured on a live process after synthesis:

phys_footprint:      13 ГБ
phys_footprint_peak: 14 ГБ

The difference between 7.7 and 13 is the MPS allocator cache, KV cache, and generation activations. On 16 GB this is tight even with one process, and ps/RSS lies here (shows tenths of a gigabyte: MPS buffers in unified memory do not show up in RSS). Measure footprint, not ps.

Consequences baked into the code:

• exactly one model lives in memory — base or instruct; switching = full reload (minutes, loudly logged);

• synthesis is serialized with a global lock — two parallel requests on 16 GB cause OOM, not speedup;

• run.sh takes a lock on the port — otherwise several clients doing autostart simultaneously each spin up their own backend copy (real case: seven processes on 16 GB);

• auto-shutdown after 1800s of idle — intentionally longer than the neighbors: restart costs ~40s of shader compilation, so keeping the process alive is more profitable;

• Whisper for auto-transcription — int8 on CPU, unloaded immediately after recognition.

Speed and four optimizations

A naive run of the upstream on M4 gave ×189 real time — 3 seconds of Ukrainian took 9.5 minutes to compute. Three fixes brought it to ×4.0, i.e. 48 times faster; the fourth brought it to ×2.7. What exactly was wrong (measured with tools/profile_steps.py and instrumentation):

1. Autocast on the AR-loop step — the main problem. The upstream hangs @torch.autocast as a decorator on _backbone_one_step, meaning the autocast region opens and closes on EVERY autoregressive step. The weight recast cache in torch lives exactly inside the region, so 1.7B fp32 parameters were converted to half every step. Backbone step: 19000 ms → 2710 ms. Now autocast is disabled by default, and the cast is done explicitly at the backbone boundary.

2. Backbone in fp32. The weights are stored in float32 (3.0B parameters = 11.4 GB), which on 16 GB means swapping. We convert only the LLM backbone to half (4.2 GB instead of 8.4), while redae and the flow decoder stay fp32 — the upstream intentionally works in full precision there, and half breaks MPS-matmul. Step: 2710 → 1387 ms.

3. Metal shader compilation. The largest part of the "slowness" turned out to be one-time: Metal compiles kernels on first execution. Individual backbone step measurements: 9873, 2104, 120, 93, 96, 97… ms. That is why the service does a warmup at startup (FIRERED_WARMUP=1) and has a long idle timeout — a restart costs ~40 seconds of compilation.

4. Reference encoding cache. generate() is called for EVERY sentence, and each call re-ran the redae encoder over the reference recording — 5.58s each time, regardless of text length. The cache key is the audio content, not the tensor id, so the cache cannot be fooled by swapping voices. On a 6.6 min audio run (14 chunks): 30 hits vs 2 misses ≈ 150s, i.e. ~11% of the time.

A minute of audio takes ~2.7 minutes to compute (Mac mini M4 / 16 GB, warm model, warm reference cache; best of 5 alternating rounds on 3.0s of Ukrainian, median gives ×2.9). Not real-time, but this is already a working tool, not "run it overnight".

Profile of warm generation: flow decoder 57%, backbone 18%, redae 11%.

n_timesteps — ODE steps in the flow decoder, the most expensive part. The parameter is available in /tts, MCP, and FIRERED_N_TIMESTEPS, but the default is 10, as in the upstream: the upstream does not document it and never suggests tuning it, and below 4 the quality noticeably coarsens. By the same measurement: 10 → ×2.7, 6 → ×1.9, 4 → ×1.3, 2 → ×1.0. Lower it only for a specific task and with an ear check.

Text normalization

The built-in TN (wetext) only knows Chinese and English, so for Ukrainian it is useless and we do not install it. 19:30, 250 грн, 2026 р. the model will voice however it happens to.

Two options:

1. write the text in words right away;

2. enable LLM-TN — FIRERED_TN_API_URL / _API_KEY / _MODEL in .env. Any OpenAI-compatible endpoint works, including a local one (llama.cpp / vLLM / Ollama) — then everything stays offline.

License

Code and weights — Apache 2.0. The upstream README separately states that zero-shot cloning is "solely for academic research purposes" — this contradicts Apache 2.0, so be careful about commercial use of cloned voices. For home use, the question does not arise.