ESP32 Voice Entry

English | 中文

A compact ESP32-S3 desk terminal that gives an agent ears, eyes, and a voice: wake word, streaming ASR, LLM, streaming TTS, barge-in, camera vision, MCP device tools, and a small paper-and-ink UI. The firmware is built from the open-source XiaoZhi ESP32 project with a board configuration and persona UI; the intelligence lives in a thin WebSocket server in this repository.

English

What it does

• Voice loop: wake word → VAD → streaming ASR → LLM → sentence-streamed TTS, with barge-in and a spoken confirmation gate for risky tools.

• Device as an MCP server: the board exposes photo, volume, brightness, theme, status, and screen tools over MCP.

• Vision: camera frames are posted to the local server and described by a local vision model.

• UI: six expression states, A/B/C information views, an idle dino strip, and a 480×320 dino game view.

• Media: agent-generated images and desktop screenshots can be shown full-screen on the device and dismissed by tap or timeout.

• OTA: the device discovers the local WebSocket endpoint through OTA.

Repository layout

src/protocol.py   Xiaozhi WS message shapes + MCP envelopes
src/audio.py      Opus codec, VAD, ASR wrappers
src/tts.py        local or cloud TTS backends
src/llm.py        OpenAI-compatible / Hermes client with streaming
src/session.py    per-connection pipeline, turn state machine, MCP calls
src/ota.py        OTA endpoint + aiohttp app
src/vision.py     camera sink, vision description, debug hooks
src/server.py     entrypoint
scripts/          fake device, MCP bridge, flashing and acceptance helpers
docs/             protocol, hardware, operations and handoff notes

Getting started (about 15 minutes)

One board from "nothing installed" to "wake it and ask the time". Everything in angle brackets is yours to fill in; the first table lists each value.

1. The values only you can supply

No working address, key or credential is stored in this repository. Every one of these is a placeholder you replace:

Download commands for all three model sets (ASR, VAD, TTS matcha + vocoder) are in docs/quickstart-single-board.md §3.3.

2. Start the server

python3.13 -m venv .venv313
.venv313/bin/pip install -r requirements.txt
cp config.example.yaml config.local.yaml        # apply the table above
export DEEPSEEK_API_KEY=<your-key>
cd src && ../.venv313/bin/python server.py --config ../config.local.yaml

The minimum edits in config.local.yaml — the defaults do not start on their own, because llm.provider defaults to a local Hermes gateway:

server:
  ws_url: ws://<server-ip>:4410
  vision_url: http://<server-ip>:4411/xiaozhi/vision
llm:
  provider: deepseek          # default "hermes" needs a gateway on 127.0.0.1:8642
tts:
  provider: volc              # or "matcha"; the matcha files must exist either way

Expected first lines: [ota] listening on http://0.0.0.0:4411/xiaozhi/ota/ and [ready] ws://0.0.0.0:4410. Ports are 4410 (voice WebSocket) and 4411 (OTA / vision / debug). Starting a second instance prints a single [fatal] port … already in use line and exits instead of crash-looping.

3. Put the firmware on the board

esptool ships with ESP-IDF, not with this repository's virtualenv, so load the toolchain first in the shell that flashes:

source <esp-idf>/export.sh

v2.5.3-persona.5 publishes merged-binary.bin, xiaozhi.bin and SHA256SUMS.txt. Pick the row that matches your board:

Prepend python -m esptool --chip esp32s3 -p <serial-port> -b 460800 to each. The public image bakes the placeholder http://rui-ota.invalid:4411/xiaozhi/ota/ (an RFC 2606 domain that never resolves), so no one's LAN address leaks — point the device at your own server with step 5.

4. Provision Wi-Fi

A board without credentials opens an open SoftAP named Xiaozhi-XXXX (the last two MAC bytes). Join it and open http://192.168.4.1/ — that is the access point's own fixed address, independent of your LAN — then pick your 2.4 GHz network and submit. If the connection times out, the board returns to this mode; re-submitting is enough, no reboot needed.

The same page has an Advanced tab (POST /advanced/submit) where you can set the OTA URL; it writes the NVS key wifi/ota_url. Use it before submitting Wi-Fi: a successful submit posts /exit, the board leaves config mode and the page is gone. There is no /advanced route of its own — open http://192.168.4.1/ and switch to the tab.

5. Point the device at your server (choose one)

• the Advanced tab above, set before submitting Wi-Fi, or

• generate a 16 KB NVS image and flash it:

  source <esp-idf>/export.sh                       # provides IDF_PATH
  python3 scripts/make_nvs_ota.py --url http://<server-ip>:4411 --out nvs-ota.bin
  python -m esptool --chip esp32s3 -p <serial-port> -b 460800 write-flash 0x9000 nvs-ota.bin

  The script stores the complete endpoint (/xiaozhi/ota/ is appended for you). Flashing 0x9000 erases NVS, so re-provision Wi-Fi afterwards.

• or build the firmware yourself with your own base (see Firmware below).

6. Talk to it and check it is alive

Say the wake word (default nihaoxiaozhi, 「你好小智」). The board connects to 4410; the server log shows [conn], [asr], [llm], [tts].

lsof -nP -iTCP:4410 -iTCP:4411 -sTCP:LISTEN          # two listeners, one pid
TOK=$(cat ~/.esp32-voice/control.token)
curl -H "Authorization: Bearer $TOK" http://127.0.0.1:4411/debug/mute

If the board never connects, look for the [ota] line: a bare base without /xiaozhi/ota/ gets a 404 and the device retries. Wake-word and audio details are in docs/quickstart-single-board.md.

Firmware

The firmware lives in a separate repository (the XiaoZhi fork) and must be built there — scripts/build.py does not exist in this repo:

# run in the firmware repository, not here
source <esp-idf>/export.sh                        # ESP-IDF release/v6.1
python3 scripts/build.py chd/chd-esp32s3-box \
  --name chd-esp32s3-box-lcd35 --language zh-CN

XIAOZHI_OTA_BASE is not expanded by this firmware tree, even though main/boards/chd/chd-esp32s3-box/config.json contains CONFIG_OTA_URL="${XIAOZHI_OTA_BASE}/xiaozhi/ota/": the build path passes the string through untouched, so a plain build bakes the literal placeholder. Set your address by rewriting the generated fragment after the last build.py run and rebuilding from a clean configure — build.py regenerates that file on every invocation, and fullclean deletes it, so the order matters:

python3 scripts/build.py chd/chd-esp32s3-box --name chd-esp32s3-box-lcd35 --language zh-CN
cp build/xiaozhi-build.sdkconfig.defaults /tmp/frag        # keep the generated options
idf.py fullclean
mkdir -p build
sed 's#${XIAOZHI_OTA_BASE}#http://<server-ip>:4411#' /tmp/frag > build/xiaozhi-build.sdkconfig.defaults
rm -f sdkconfig
idf.py -DIDF_TARGET=esp32s3 \
  -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;build/xiaozhi-build.sdkconfig.defaults" \
  -DBOARD_NAME=chd-esp32s3-box-lcd35 reconfigure
idf.py build && idf.py merge-bin
strings -a build/xiaozhi.bin | grep -c 'XIAOZHI_OTA_BASE'      # must print 0

Or leave the image alone and write the NVS key wifi/ota_url (step 5). The full recipe and partition table are in docs/firmware-release.md.

Release assets: v2.5.3-persona.5 ships both merged-binary.bin (full image at 0x0, erases NVS) and xiaozhi.bin (app at 0x20000), plus a SHA-256 manifest. Older tags are gone — the repository history was rebuilt on 2026-10-30. For an existing board flash the app only; for a new board use the merged image or build once from source. Step-by-step: docs/quickstart-single-board.md.

Known limitations

• Validated on one ESP32-S3 board and 2.4 GHz Wi-Fi.

• The dino game view measures about 25 fps; the 30 fps target is not reached.

• Desktop capture requires the host's screen-recording permission.

• The project is a desk device, not a far-field speaker.

Acknowledgements

This project stands on open-source work:

• 78/xiaozhi-esp32 — firmware base, board framework, protocol and MCP foundations.

• ESP-IDF, LVGL, ESP-SR — embedded runtime, UI and wake-word engine.

• sherpa-onnx and Opus — local speech recognition and audio codec.

• The XiaoZhi community and the open-source projects that documented the protocol, MCP patterns and device workflows.

Thanks to the maintainers and contributors of these projects.

License

MIT — see LICENSE. Copyright belongs to the esp32-voice-entry contributors. Upstream components retain their own licenses and attribution.

Related MCP server: termius-mcp

中文

项目简介

一个桌面级 ESP32-S3 语音终端：唤醒词、流式 ASR、LLM、流式 TTS、打断、摄像头 视觉、MCP 设备工具，以及一套纸墨风格的小屏 UI。固件基于开源的小智 ESP32 项目，加上板级配置和人格化界面；真正的智能逻辑在本仓库的轻量 WebSocket 服务端。

主要能力

• 语音闭环：唤醒 → VAD → 流式 ASR → LLM → 按句流式 TTS，支持打断； 高风险工具调用前有语音确认。

• 设备即 MCP 服务：拍照、音量、亮度、主题、状态、屏幕等工具通过 MCP 暴露。

• 视觉：摄像头画面上传到本机服务，由本地视觉模型描述。

• 界面：六种表情状态、A/B/C 信息视图、空闲恐龙条、480×320 恐龙游戏视图。

• 媒体：Agent 生成的图片和桌面截图可在设备上全屏显示，点击或超时关闭。

• OTA：设备通过 OTA 获取本机 WebSocket 地址。

使用指导书（约 15 分钟）

从"什么都没装"到"唤醒它并问一句时间"。尖括号里的内容都要你自己填， 第一张表就是清单。

1. 只有你能提供的信息

仓库里没有任何可用的地址、密钥或凭据；下面每一项都是占位符，需要你替换：

三套模型（ASR、VAD、TTS 的 matcha + 声码器）的下载命令都在 docs/quickstart-single-board.md §3.3。

2. 启动服务端

python3.13 -m venv .venv313
.venv313/bin/pip install -r requirements.txt
cp config.example.yaml config.local.yaml        # 按上表逐项改
export DEEPSEEK_API_KEY=<your-key>
cd src && ../.venv313/bin/python server.py --config ../config.local.yaml

config.local.yaml 里至少要改这几项——默认配置不能直接跑，因为 llm.provider 默认指向本地 Hermes 网关：

server:
  ws_url: ws://<server-ip>:4410
  vision_url: http://<server-ip>:4411/xiaozhi/vision
llm:
  provider: deepseek          # 默认的 hermes 需要 127.0.0.1:8642 上的网关
tts:
  provider: volc              # 或 matcha；两条 matcha 模型路径都必须存在

启动后应看到 [ota] listening on http://0.0.0.0:4411/xiaozhi/ota/ 与 [ready] ws://0.0.0.0:4410。端口：4410 语音 WebSocket、4411 OTA / 视觉 / 调试。再起第二个实例只会打印一行 [fatal] port … already in use 然后退出，不会再刷 Traceback。

3. 把固件刷到板上

esptool 随 ESP-IDF 提供，本仓库的虚拟环境里没有，所以刷机前先在同一个 shell 里加载工具链：

source <esp-idf>/export.sh

v2.5.3-persona.5 提供 merged-binary.bin、xiaozhi.bin 与 SHA256SUMS.txt。 按下表选一行：

每条前面加 python -m esptool --chip esp32s3 -p <serial-port> -b 460800。 公开镜像里编的是占位地址 http://rui-ota.invalid:4411/xiaozhi/ota/（RFC 2606 保留域名，永不解析），所以不会泄漏任何人的内网地址——用它自己的服务器请走第 5 步。

4. 配网

没有保存 Wi-Fi 的板子会开一个开放热点，名为 Xiaozhi-XXXX（X 为 MAC 末两字节）。 连上后打开 http://192.168.4.1/（这是热点自己的固定地址，与你的局域网网段无关）， 选你家 2.4 GHz 网络并提交。连接超时会自动回到这个模式，重新提交即可，不用重启。

同一个页面上有个 Advanced 标签（表单 POST /advanced/submit），可以填 OTA 地址， 会写进 NVS 的 wifi/ota_url。一定要在提交 Wi-Fi 之前填：提交成功后页面会 POST /exit、 设备退出配网模式，这个页面就没了。并不存在 /advanced 这个独立地址——打开 http://192.168.4.1/ 再切到标签页。

5. 让设备指向你的服务器（三选一）

• 配网页的 Advanced 标签（要在提交 Wi-Fi 之前填）；

• 生成 16 KB 的 NVS 镜像再刷：

  source <esp-idf>/export.sh                       # 提供 IDF_PATH
  python3 scripts/make_nvs_ota.py --url http://<server-ip>:4411 --out nvs-ota.bin
  python -m esptool --chip esp32s3 -p <serial-port> -b 460800 write-flash 0x9000 nvs-ota.bin

  脚本会自动补成完整端点（/xiaozhi/ota/）；刷 0x9000 会清 NVS，刷完要重新配网；

• 或者自己重编固件、把自己的地址编进去（见下面「固件」）。

6. 跟它说话并确认链路

说唤醒词（默认 nihaoxiaozhi，即「你好小智」）。板子会连上 4410，服务端日志依次出现 [conn]、[asr]、[llm]、[tts]。

lsof -nP -iTCP:4410 -iTCP:4411 -sTCP:LISTEN          # 两个监听，同一个 pid
TOK=$(cat ~/.esp32-voice/control.token)
curl -H "Authorization: Bearer $TOK" http://127.0.0.1:4411/debug/mute

如果设备始终不连：先看 [ota] 那行——只写了基址、缺 /xiaozhi/ota/ 会 404 并让设备 一直重试。唤醒词与音频细节见 docs/quickstart-single-board.md。

固件

固件在另一个仓库（小智 ESP32 的 fork）里，必须到那边构建——本仓库没有 scripts/build.py：

# 在固件仓库执行，不是本仓库
source <esp-idf>/export.sh                        # ESP-IDF release/v6.1
python3 scripts/build.py chd/chd-esp32s3-box \
  --name chd-esp32s3-box-lcd35 --language zh-CN

这棵固件树不会展开 XIAOZHI_OTA_BASE：虽然 main/boards/chd/chd-esp32s3-box/config.json 里写着 CONFIG_OTA_URL="${XIAOZHI_OTA_BASE}/xiaozhi/ota/"，构建路径是原样透传的， 直接编出来就是字面占位符。要改地址，必须在最后一次 build.py 之后重写生成的 fragment、再从干净配置重编—— build.py 每次运行都会重写这个文件，而 fullclean 又会把它删掉，所以顺序不能错：

python3 scripts/build.py chd/chd-esp32s3-box --name chd-esp32s3-box-lcd35 --language zh-CN
cp build/xiaozhi-build.sdkconfig.defaults /tmp/frag        # 先保存生成的选项
idf.py fullclean
mkdir -p build
sed 's#${XIAOZHI_OTA_BASE}#http://<server-ip>:4411#' /tmp/frag > build/xiaozhi-build.sdkconfig.defaults
rm -f sdkconfig
idf.py -DIDF_TARGET=esp32s3 \
  -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;build/xiaozhi-build.sdkconfig.defaults" \
  -DBOARD_NAME=chd-esp32s3-box-lcd35 reconfigure
idf.py build && idf.py merge-bin
strings -a build/xiaozhi.bin | grep -c 'XIAOZHI_OTA_BASE'      # 必须是 0

或者不动镜像，改写 NVS 的 wifi/ota_url（第 5 步）。完整配方与分区表见 docs/firmware-release.md。

发布资产：v2.5.3-persona.5 同时提供 merged-binary.bin（整包，烧 0x0， 会清 NVS）与 xiaozhi.bin（app，烧 0x20000），外加 SHA-256 清单。更早的 tag 已不存在——仓库历史在 2026-10-30 重建过。已有同款板升级只烧 app；新板用整包 或从源码构建一次。完整步骤见 docs/quickstart-single-board.md。

已知限制

• 目前只在一块 ESP32-S3 板和 2.4 GHz Wi-Fi 上验证。

• 恐龙游戏视图约 25 fps，尚未达到 30 fps。

• 桌面截图需要宿主系统授予屏幕录制权限。

• 定位是桌面设备，不是远场音箱。

致谢

本项目建立在以下开源工作之上：

• 78/xiaozhi-esp32：固件基础、板级框架、 协议与 MCP 基础。

• ESP-IDF、LVGL、 ESP-SR：嵌入式运行时、UI 与唤醒词引擎。

• sherpa-onnx、 Opus：本地语音识别与音频编解码。

• 小智社区，以及记录协议、MCP 模式和设备工作流的开源项目。

感谢这些项目的维护者和贡献者。

许可证

MIT，见 LICENSE。版权归 esp32-voice-entry 贡献者所有；上游组件保留 各自的许可证与署名。