esp32-pet
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@esp32-petset the pet to happy and have it say hello"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
BIP
A desk pet on the Waveshare ESP32-S3-LCD-1.47B: a 172×320 face, one RGB LED, and a USB link a computer can drive. Cursor, Claude, and Codex connect through the built-in MCP server.
The board is ESP32-S3R8, 16 MB flash, 8 MB PSRAM, ST7789, QMI8658 IMU, and an 18-pin header for add-ons. Tilting the board turns the face. The only onboard key is BOOT (GPIO0). Hold BOOT while resetting to enter the ROM download mode.
Flash
Firmware builds in Docker with ESP-IDF 5.5.1. The image is pulled on first use. Output stays in .deps/, which is not part of the source tree.
bash scripts/build.sh
pip install esptool # or: python3 -m venv .venv && .venv/bin/pip install esptool
bash scripts/flash.sh /dev/cu.usbmodem1101On Linux the port is usually /dev/ttyACM0.
USB is enough. Wi-Fi is optional and only for the UDP state packet:
python3 -m venv .venv
.venv/bin/pip install -e .
.venv/bin/python scripts/provision.py --port /dev/cu.usbmodem1101 --ssid "Your2.4GHzSsid"The password is written to the board and is not kept in the repo. The pairing file is .deps/pet-device.json (mode 600).
Related MCP server: obsbot-mcp
MCP
Install the bridge once:
python3 -m venv .venv
.venv/bin/pip install -e .This checkout already has a project config at .cursor/mcp.json. Restart Cursor and the esp32-pet server is available. Tools:
Tool | What it does |
| Face, gaze, mouth, blink |
| Timed face keyframes |
| Speak on the Mac with |
| Port, last face, recent button presses |
Set PET_USB_PORT when more than one USB serial device is plugged in.
Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json):
{
"mcpServers": {
"esp32-pet": {
"command": "/absolute/path/to/esp32-pet/.venv/bin/python",
"args": ["-m", "bip_bridge.mcp_server"],
"cwd": "/absolute/path/to/esp32-pet"
}
}
}Codex (~/.codex/config.toml):
[mcp_servers.esp32-pet]
command = "/absolute/path/to/esp32-pet/.venv/bin/python"
args = ["-m", "bip_bridge.mcp_server"]
cwd = "/absolute/path/to/esp32-pet"The server speaks MCP over stdio. It prints protocol messages on stdout and nothing else.
Voice and the local page
Voice needs ElevenLabs. Copy .env.example to .env and fill in the key and agent id, then:
.venv/bin/pip install -e ".[voice]"
pnpm live
pnpm live -- --ptt # BOOT (GPIO0) is the talk key
pnpm demo # macOS say, no API key
pnpm dev # dashboard at http://127.0.0.1:5173pnpm dev needs Node 22 and pnpm install. The page is a host preview of the face plus buttons for the onboard key and three add-on slots.
Pins
Use | GPIO |
LCD MOSI, SCLK, CS, DC, RST, backlight | 45, 40, 42, 41, 39, 46 |
RGB LED | 38 |
IMU SDA, SCL | 48, 47 |
TF card CMD, SCK, D0, D1, D2, D3 | 15, 14, 16, 18, 17, 21 |
USB D−, D+ | 19, 20 |
BOOT | 0 |
Free for add-ons: GPIO1, GPIO2, GPIO4–GPIO13. GPIO14–GPIO18 and GPIO21 are the TF slot. Leave them if the card should keep working.
Add-ons
An add-on is one C file and one line in firmware/pet/main/addon.c. firmware/pet/addons/example_button.c is a button on GPIO2 that reports button 4 (cycles the LED). It is not compiled until you add it to SRCS in firmware/pet/main/CMakeLists.txt and register &example_button_addon.
pet_emit_button(n) reuses the face actions: 1 pet, 2 cycle mood, 3 agent, 4 LED mode. The onboard BOOT key is separate. It always cycles the mood and is reported to the host as button 1.
The host hears one line per press:
PET_BTN button=1 mood=HAPPY led=0 energy=95A JSON line on the same serial port sets the face (mood, gazeX, gazeY, flags, energy, title, subtitle). STATUS answers with PET_STATUS and PET_IP.
Layout
firmware/pet/ ESP-IDF project for this board
bip_bridge/ USB controller and the MCP server
scripts/ build, flash, provision, voice
src/ optional local dashboard.env, .venv, .deps, and node_modules stay on your machine.
Available Tools
4 toolsbip_get_statusA
Return BIP's current expression state, USB port, Wi-Fi IP, and recent physical button presses.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are supplied, so the description carries the full burden. 'Return' strongly implies a non-mutating read, and the field list tells the agent what data comes back, but there is no explicit statement that it has no side effects, no note on freshness or whether repeated polling is acceptable, and no cost/latency context. Useful but not a full behavioral picture for an unannotated tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence, front-loaded with the verb and immediately enumerating the returned data. Every clause carries information and there is no filler or repetition of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description must describe return values, and it does so by naming the four categories of status data. It could go further on the shape of those values (e.g., what a button-press record looks like, IP format), but for a simple status read this is largely sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing for the description to disambiguate; the baseline for a no-param tool applies. Nothing in the description misrepresents or omits parameter behavior because none exists.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb ('Return') plus an enumerated set of returned fields: expression state, USB port, Wi-Fi IP, and recent button presses. That is far more concrete than a bare 'get status', and the read-only nature is implicit against siblings like bip_set_state and bip_speak. It stops short of explicitly naming the sibling it complements, which keeps it from a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: an agent can infer this is the read counterpart to bip_set_state, but the description never says when to call it (e.g., before mutating state to check current conditions) or what alternatives exist. For a zero-parameter getter the inference burden is low, so a 3 rather than a 2.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
bip_play_timelineC
Play a sequence of time-coded facial animation keyframes on BIP (non-blocking firmware loop).
| Name | Required | Description | Default |
|---|---|---|---|
| keyframes | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden, and it only discloses that playback is non-blocking. It does not say whether the timeline replaces the current facial state, what happens if playback is already running, whether keyframe times are relative to invocation, or how errors and completion are surfaced.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One tightly written sentence with no filler, and the essential purpose is front-loaded. Its brevity is efficient but contributes to the missing detail elsewhere.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This is a structurally complex tool — a required array of keyframe objects — with no annotations and no output schema, so the description should explain keyframe field meaning and playback lifecycle. Instead it covers only the high-level intent and one execution trait, leaving an agent unable to construct a valid, meaningful keyframe list confidently.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single keyframes parameter has 0% schema description coverage and a nested object with six undocumented subfields (at_ms, state, blink, gaze_x, gaze_y, speaking). The phrase "time-coded" implies at_ms semantics, but state values, gaze coordinate ranges, and units are left entirely unexplained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ("Play") and a specific resource ("a sequence of time-coded facial animation keyframes on BIP"), so the agent understands this schedules an animation timeline rather than a one-shot state change. It stops short of explicitly contrasting itself with bip_set_state or bip_speak, so the differentiation is inferred rather than stated.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to choose a timeline playback over bip_set_state or bip_speak, and no mention of prerequisites such as connecting to the device first. The parenthetical about a non-blocking firmware loop is the only hint about invocation context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
bip_set_stateB
Set BIP's physical facial expression, eye gaze direction, speaking mouth animation, or trigger an expressive blink over USB Serial.
| Name | Required | Description | Default |
|---|---|---|---|
| blink | No | Force an immediate blink. | |
| state | Yes | Target expression state for BIP. | |
| gaze_x | No | Horizontal gaze (-100 left .. 100 right). | |
| gaze_y | No | Vertical gaze (-100 up .. 100 down). | |
| speaking | No | Whether the mouth talking animation is active. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does supply useful context that this drives physical hardware over USB Serial. It omits what happens when the device is disconnected, whether the set state persists until changed, and how gaze/speaking interact with the state enum, which matters for a stateful hardware mutation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence that enumerates the controllable surfaces with zero filler. It is slightly list-like and could be tightened, but nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a five-parameter hardware-control tool with no output schema, the description covers the what but not the interaction model: it never clarifies whether the 'blink' boolean and the 'blink' state value are equivalent or whether gaze/speaking flags override the state. That ambiguity is the main gap an agent would hit in practice.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so ranges, the state enum, and each field's meaning are already documented. The description merely summarizes those parameters and adds no precedence or combination rules beyond the schema, making the baseline 3 appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Set') and the resource (BIP's facial expression, gaze, speaking animation, blink) and even names the transport (USB Serial). It never references the siblings, so an agent must infer that bip_speak is the text-to-speech counterpart while this only drives the mouth animation, but the operation itself is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the enumerated capabilities rather than stated: nothing says when to prefer this over bip_speak or bip_play_timeline, nor whether animations from bip_play_timeline take precedence. An agent can guess the intent, but the choice among siblings is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
bip_speakC
Speak English text aloud on Mac and synchronize BIP's mouth, gaze, and blinks to the audio duration.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | Text for BIP to speak. | |
| blink | No | ||
| gaze_x | No | ||
| gaze_y | No | ||
| emotion | No | happy |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It discloses that audio is played and animation is synchronized to the audio duration, but never states whether the call blocks until speech completes, whether it can be interrupted, what happens if audio fails, or what platform prerequisites exist (English only, Mac only is mentioned but not why).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence that leads with the primary action and appends the animation behavior. No filler, though the two ideas (speech and rig synchronization) are compressed into one line without structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-parameter mutation-style tool with no annotations, no output schema, and 80% of parameters undocumented, the description leaves too much unknown. The most decision-relevant fact for an agent — whether the call waits for the audio to finish — is absent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 20%, so the description needs to compensate. It does imply that gaze and blink are driven by this tool, giving indirect meaning to blink/gaze_x/gaze_y, but it never explains the coordinate ranges, units, or defaults for the gaze integers, nor what the emotion choices do to the delivery.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a concrete verb and resource ("Speak English text aloud on Mac") plus the distinctive side effect of lip/gaze/blink synchronization, which clearly separates it from bip_get_status or bip_set_state. It stops short of naming a sibling or stating how it differs from bip_play_timeline, which could also drive animation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no explicit when-to-use or when-not-to-use guidance and no mention of alternatives among bip_set_state, bip_play_timeline, or bip_get_status. The agent must infer that this is the tool for voiced dialogue rather than for silent posing or timeline playback.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
4 tool updates
v0.1.0- First observed
bip_get_status - First observed
bip_play_timeline - First observed
bip_set_state - First observed
bip_speak
TDQS
Scored across 4 tools
Each tool targets a distinguishable action: setting a single state, playing a keyframe timeline, reading status, and speaking text. There is mild overlap since bip_set_state also controls speaking mouth animation while bip_speak synchronizes mouth/gaze/blink, so an agent might hesitate over which to use for mouth motion, but the descriptions clarify the intended split.
All four tools use a consistent snake_case pattern with the shared 'bip_' prefix and a verb-led name (set_state, play_timeline, get_status, speak). The convention is predictable and readable across the whole surface.
Four tools is on the lean side but each covers a distinct facet (state, sequence playback, status, speech) of driving a physical pet device. It is slightly thin — no separate stop/reset tool — but nothing feels redundant.
The core actions (set expression, play animation, speak, read status) are covered, but there is no way to stop or interrupt a running timeline, reset to a known idle state, or enumerate valid expressions/gazes. These gaps will force agents to improvise when recovering from an animation or discovering valid state values.
Maintenance
Related MCP Connectors
Turns a phone into a camera+Bluetooth remote so AI assistants can see and control any PC.
- DNS-DATA RemoteDeprecatedcom.dns-data
Connect AI assistants to authorized Windows and macOS devices for files, commands, and management.
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Send and read OpenPhone messages, calls, contacts and phone numbers.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to control a Live2D desktop pet's expressions and actions via MCP protocol.MIT
- AlicenseBqualityAmaintenanceControls an OBSBOT Tiny 2 camera gimbal, zoom, and wake/sleep over UVC/USB without any vendor SDK.80106 npm2MIT
- AlicenseNot gradedqualityAmaintenanceEnables MCP-compatible agents to control a desktop mascot by switching its appearance and voice, showing speech-bubble dialogue, and managing state such as affinity.12 npm1MIT
- AlicenseBqualityCmaintenanceEnables natural-language control of DK-META2 devices through local BLE for vibration, linear extension, and rotation. It also supports scanning, connecting, status checks, and safe stop/disconnect operations without cloud access.81MIT