# AI Hardware Lab — OkabeBot

A push-to-talk Codex companion with a talking pixel character and MCP tools for real hardware. Hold a button, speak, and release. Local speech recognition transcribes the recording; Codex can answer aloud, change expressions, show a picture, use a camera, or press a SwitchBot.

The default character is an original pixel-art interpretation of **Rintaro Okabe from Steins;Gate**: dark hair, lab coat, expressive eyes, and an audio-driven mouth. This is an unofficial fan project, with no affiliation or endorsement. The artwork is drawn in code; no anime frames or actor voice recordings are included. Pip, the original robot face, remains an option.

![Okabe character](docs/okabe.png)

## See the integrations

![Okabe presenting integration examples](docs/integration-gallery.png)

Illustrative views generated by `python scripts/preview-integrations.py`: GitHub
notifications, upcoming calendar events, grouped door/Ring activity, music,
TV/sports reactions, Codex completion, and focus timers. Alexa's timer card is a
preview; account import and live validation are separate setup steps. These are sample
messages, not screenshots of private account activity.

## One controller, flexible hardware

| Setup | Controller | Display / camera / Bluetooth | Microphone / speaker / button |
| --- | --- | --- | --- |
| One Pi | Same capable Linux Pi | Local service | Local service |
| Multiple Pis | Mac or Linux computer | SSH to a Pi | SSH to another Pi |
| Hybrid | Mac, Linux computer, or capable Pi | Local or SSH | Local or SSH |

Each device endpoint independently selects `local` or `ssh`. Both use the same loopback HTTP protocol. SSH connections use your existing verified host keys; no hardware service needs an exposed network port.

The existing deployment uses a Mac controller and one legacy Pi Zero WH satellite
combining display, camera, Bluetooth, audio and button services. **The all-in-one ARM64 setup is an architectural option, not yet hardware-validated.** A Pi Zero cannot run this controller stack. Use a modern 64-bit Pi with sufficient RAM and supported dependencies for local Whisper and Codex CLI. Codex model inference still requires the OpenAI service; “local” refers to transcription, speech synthesis, and tool execution.

## Quick start

1. Install Python 3.10+, a signed-in [Codex CLI](https://github.com/openai/codex), and the platform speech dependencies: macOS `say`/`afconvert` are built in; Linux needs `espeak-ng` and `ffmpeg`.
2. Install the controller from this checkout:

   ```sh
   python3 -m venv .venv
   . .venv/bin/activate
   pip install -e '.[voice,cast,notifications]'
   cp examples/single-pi.json okabebot.local.json
   # Or copy examples/multi-pi.json for SSH-connected devices.
   ```

3. Configure and start the Pi services using [device setup](docs/devices.md). Edit `okabebot.local.json` with the matching ports and local/SSH endpoints. Relative paths resolve from the config file, so copy the example to the project root before editing.
4. Download the configured transcription model once, then start:

   ```sh
   okabebot-model
   okabebot
   # Alternative: okabebot --config /absolute/path/my-config.json
   ```

5. Open the loopback control URL printed in the terminal. It contains a private access token. Hold the hardware button while speaking, release to submit (20-second maximum). The page shows transcripts, tool activity, and any remaining approval prompts. Stop the controller with Ctrl-C.

The controller resumes its saved Codex chat. The button disarms while Codex works or speaks. A renewable lease disarms it if the controller disconnects. Recordings are deduplicated before submission; uncertain actions are not automatically replayed.

For supervised Mac startup, run **`./scripts/start-mac.sh`** from this checkout.
It starts the configured voice worker, waits for its warm-up, and starts the
controller. Child failures restart with backoff; unavailable Pis reconnect without
re-enabling a manually paused microphone. The dashboard shows device and voice
readiness. The control URL stays valid across restarts; its private credential is
stored in `state/controller-token`. Delete that file while stopped to rotate it.

To start after Mac login and recover if the supervisor exits, stop the foreground
process, then run `.venv/bin/python scripts/install-mac.py`. This makes a private
installation in `~/Library/Application Support/OkabeBot` and installs its login
service. It copies the prepared environments, settings, reference manifest/audio,
models and persistent application state, without copying development scratch data.
This avoids macOS background-process restrictions on Documents/Desktop/Downloads.
The installer refuses an existing destination rather than overwriting private
state. For updates, stop the printed service, back up that installation, and update
its code/environments deliberately; do not overwrite its `.local` state. The copied
environments are for this same Mac, not portable to another machine.

The installed one-command start script is
`~/Library/Application Support/OkabeBot/scripts/start-mac.sh`. To use the login
service instead of foreground mode, run that script with `--install` (it also
restarts the service). This is a per-user LaunchAgent, not a system daemon: log
into the Mac after a reboot. It prints the service label and its stop command. Logs and supervisor status live in `state/runtime/`; the private
dashboard URL is also in `state/ptt/control.json`. Stop an independently running
controller/worker before the first supervised launch. Foreground mode stops both
children with Ctrl-C. Prepared Linux hosts can also run
`python -m okabebot.runtime --config /path/to/config.json` under their service manager.

## Auto-approval

Hardware MCP tool calls are **automatically approved by default** in this voice controller. This includes physical button presses, camera capture, and audio recording when requested by Codex. The policy is limited to the `hardware-lab` server and the named `approvals.allowed_tools` list. It does not approve shell commands, file edits, other MCP servers, arbitrary forms, or explicit review denials.

To require manual confirmation, set:

```json
{
  "approvals": { "auto_approve_hardware": false }
}
```

Or keep auto-approval enabled and remove individual tools from `allowed_tools`. Restart the controller after editing configuration. Automatic decisions are shown in the control page. Unknown request formats fall back to manual approval. This setting belongs to this controller, not to every Codex chat or the desktop app.

## Make the character yours

Configuration is merged with defaults, so a small override is sufficient:

```json
{
  "character": {
    "name": "Okabe",
    "style": "okabe",
    "default_voice": "calm",
    "intensity": 0.6,
    "palette": {
      "background": "#101e25",
      "shell": "#accdc1",
      "accent": "#a5f4ce"
    }
  }
}
```

The control page’s **Character** selector switches between Okabe, Makise Kurisu, Pip, and your configuration default. It saves the selection in `state/profile.json`; both the controller and MCP process read it for subsequent turns. Changing characters starts a fresh chat on the next turn so the previous persona cannot override the new one; previous chats remain in Codex history. Switching is disabled during recording or a Codex turn. A saved voice override stays selected; otherwise Kurisu and Pip use the warm preset. Update the display Pi’s service and renderer files together when upgrading: older services do not recognize `kurisu`.

Set `style` to `pip`, `name` to `Pip`, and `persona` to a friendly robot description for the original robot. The `character.persona` text controls the conversational character independently of its drawing; current preferences are included with every voice turn, including resumed chats. The `shell` color applies to Pip's body. The character supports neutral, happy, curious, concerned, and excited emotions, with idle, listening, thinking, and speaking activities. Mouth movement follows the synthesized audio amplitude rather than phoneme-level lip sync.

Voice presets are `warm`, `calm`, and `robotic`. Map them to installed voices in `speech.mac_voices` or `speech.linux_voices`. `speech.engine: auto` selects macOS `say` or Linux eSpeak. These are generic system voices, not the character's actor. A voice selected through `set_character_voice` is saved in `state/character.json` and takes precedence over `default_voice`; remove that saved preference to use the config default again. Tool argument `voice: default` uses the selected voice (`pip` remains a compatibility alias).

For streaming Qwen character voices on Apple Silicon, create a separate environment
and install `mlx-audio==0.6.1`. Keep the reference manifest and audio in ignored
private state. Each manifest key is a character style, containing `audio` (relative
to the manifest) and `text` (its exact English transcript). References and voice
samples are not included. Configure the supervisor with argv, without a shell:

```json
{"speech": {
  "engine": "local",
  "local_socket": "state/tts.sock",
  "worker_command": [".voice-venv/bin/python", "scripts/qwen-voice-worker.py",
    "--manifest", "state/voices/references.json", "--socket", "state/tts.sock",
    "--revision", "0d6bb6fe33f92d47a507e23b9148940e8366ab5b"],
  "worker_ready_timeout_seconds": 120
}}
```

Worker arguments run relative to the config file's directory. Its Python environment
is separate from the controller's, and the current MLX worker requires Apple Silicon.
The worker keeps references warm and streams 24 kHz audio through the Pi playback
and animation path. Delivery follows the reference performance; emotion instructions
and voice presets do not change this model's delivery. Missing character references
are shown as unavailable; select **Local speech** in Dev to use the system voice.
`speech.worker_command` can launch any worker implementing the protocol documented
in [adapter contracts](docs/adapters.md), keeping model dependencies out of the controller.

The Codex instruction interprets “wave” as a happy face plus **one SwitchBot press**. A durable request ID prevents a retry from pressing again. Different physical gestures can be implemented by changing the instruction and tool adapter together.

## Response timing and voice engines

The control page reports transcription, first model message, total Codex turn,
and speech synthesis/stream timing. These overlap: total turn time includes
hardware calls and playback. Playback launch is a device-clock estimate, not an
acoustic measurement. Hardware RPC calls reuse per-process SSH tunnels and never
automatically replay failed actions. Restart the controller after a tunnel fails.

`speech.sync_lead_seconds` defaults to 0.5 seconds; increase it if audio and the
face drift. `transcription.beam_size` (1–5, default 5) and `codex.effort`
(`low`, `medium`, `high`; default `low`) allow latency/quality comparisons.

Local speech remains the default. Optional `openai` speech streams 24 kHz PCM to
the voice Pi while updating the mouth envelope. Update both Pi services and
include `pi/stream_audio.py` when upgrading. Streams are capped at 30 seconds;
an abandoned stream releases the speaker after a timeout. No automatic fallback
or replay occurs after partial playback.

To enable cloud voices, configure `OPENAI_API_KEY` in the controller service's
private environment and restart it. The web engine selector stays disabled until
a key is configured. Codex ChatGPT sign-in does not provide speech API access.
For the `okabebot` systemd user service, run
`python3 scripts/configure-cloud-voice.py` in a private terminal on the controller.
It prompts without echoing, saves the key in ignored `state/openai.env` with
permissions 600, configures the service, and restarts it. Keep that state
directory and service configuration on persistent storage for a live USB.
Do not put the key in configuration JSON, a command argument, or Git.

Cloud reply text is sent to OpenAI and billed through the API. Configure
`speech.cloud_model` and `speech.cloud_voices` (warm/calm/robotic) to choose the
model and voices. The default is `gpt-4o-mini-tts`, with coral/onyx/alloy voices.
These are synthetic voices, not the characters' actors. Cloud speech requires a
live API test before its sound quality and latency can be compared; local tests
only verify the streaming transport and error handling.

## MCP tools

Run `okabebot-mcp` as a stdio MCP server. The voice controller registers it automatically for its App Server connection. Other MCP clients can launch it with `OKABEBOT_CONFIG` pointing at an absolute config file.

| Area | Tools |
| --- | --- |
| SwitchBot | `get_bot_status`, `press_bot` |
| Screen | `display_message`, `display_image`, `clear_display`, `get_display_status`, `preview_display` |
| Camera | `capture_camera`, `record_camera` |
| Audio and button | `get_voice_status`, `get_button_events`, `record_audio`, `play_audio`, `speak` |
| Character | `character_speak`, `character_expression`, `list_character_voices`, `set_character_voice` |

Hardware is optional per operation: an unavailable camera or Bot makes that tool fail, not a substitute device act. The voice workflow needs both running services. The display service currently needs a graphical desktop session even if you only use its camera/Bluetooth tools.

For a Linux PC controller, remote control-page access, and live-USB persistence
limitations, see [Linux controller setup](docs/linux-controller.md).

## Development

```sh
pip install -e '.[dev]'
ruff check .
ruff format --check .
OKABEBOT_CONFIG=examples/single-pi.json python -m unittest discover -s tests -v
python -m build
```

Unit tests use mocks and do not actuate hardware. The separate `tests/*_smoke.py` scripts are opt-in live checks: some change the screen, capture media, or play speech. `mcp_smoke.py` only presses the Bot when explicitly passed `--press`.

- `okabebot/`: controller, MCP tools, configuration, speech, transport, approvals, and control page.
- `pi/`: Python 3.7-compatible hardware services and character renderers.
- `examples/`: public, credential-free topology/configuration examples.
- `tests/`: offline tests and explicitly invoked live checks.

Local configs (`*.local.json`), state, SSH keys, recordings, and transcripts are ignored. Keep private material outside source control. Hardware services bind only to loopback and trust local processes; they are not intended for untrusted multi-user hosts. The control page is loopback-only and token protected. For a remote controller, forward its port through SSH rather than exposing it publicly.

Code is MIT licensed. Bundled DejaVu fonts retain their own license in `okabebot/assets/LICENSE-DejaVu.txt`. Steins;Gate and its characters belong to their respective rights holders; the code license does not grant rights to those characters.

## Notifications and tablet controller

A tablet-friendly companion view offers large character portraits and an inbox, with transcription and diagnostics behind Dev. Notifications appear as cards held by the animated character in the bottom of the Pi screen. Important items can use local speech when the conversation is idle. GitHub account notifications and deployments, plus Google Calendar via private iCal feed, are optional sources. See [notification setup and API](docs/notifications.md) and [Codex completion cards with a chime](docs/codex-notifications.md).

Spotify playback can animate the character with music notes and a scrolling song footer. See [Spotify setup](docs/spotify.md) and the [Home Assistant, Tuya, Ring, and Alexa plan](docs/home-assistant.md).

## Guided setup and extending integrations

Codex users can invoke `$okabebot-setup` from this checkout; its instructions live
in [.agents/skills/okabebot-setup](.agents/skills/okabebot-setup/SKILL.md). It covers
personal configuration, single/multiple devices, optional providers and migration
without assuming this author's accounts or network.

On prepared Linux, `./scripts/start-linux.sh --bootstrap` installs the app extras,
caches the speech model and starts; subsequent launches use
`./scripts/start-linux.sh`. Read [Linux setup](docs/linux-controller.md) for the
required OS/account steps and live-USB limitations.

See [adapter contracts](docs/adapters.md) to add media or notification sources.
Provider code returns normalized state; presentation, sound, grouping and device
transport remain separate.

## Commit documentation check

Run `./scripts/install-hooks.sh` once per checkout to enable the tracked commit
hook. Every commit must either stage a README update or include a trailer such as
`README-Reviewed: Internal fix; documented setup and behavior remain accurate.`
Review the README before using that trailer. The hook requires review evidence;
it cannot determine whether prose is correct or automatically rewrite it.

Context-aware Echo speaking has an optional Home Assistant package with a
disabled-by-default toggle, dry-run routing, and a cooldown; see
[Alexa setup](docs/alexa.md#optional-context-aware-speaking).

For direct ALSA playback, configure the physical card's speaker mixer rather
than the default virtual Master; see [volume setup](docs/devices.md#match-the-volume-mixer-to-the-playback-device).

The tablet now includes **Your day**: remaining calendar appointments, a next-event
countdown and a silent agenda card on the Pi. Optional scheduled morning cards and
signed GitHub workflow/deployment webhooks are documented in
[notification setup](docs/notifications.md). Webhooks require a private secret,
explicit repository list and separately configured HTTPS ingress. Set
`notifications.github_polling: false` after registering hooks for webhook-only delivery.
[TV classification](docs/tv.md) now covers common video formats, provider genres and
cached, verified Wikidata movie/series genre enrichment, with neutral fallbacks.

Optional [AWS webhook relay](docs/notifications.md#aws-relay-outbound-only-mac-connection)
uses Lambda HTTPS → SQS → an outbound Mac consumer, preserving signed delivery and
local deduplication while the Mac is disconnected. Deployment/connect scripts are
in `scripts/`; the source AWS login must remain valid. No permanent AWS keys or
public dashboard access are required. Team crests also support verified Wikidata
abbreviations and independent partial enrichment when one opponent is unresolved.

The Pi layout uses larger notification/media text, 80-pixel sports crests and a
narrower Pip with clear side lanes. Working/deployment activity uses a subtle
thinking expression, without loading dots beneath the media footer. The live
canvas and preview share native 480×320 layout geometry. Identity Center users can
connect the relay with `--consumer-profile PROFILE`; a queue-only permission-set
template is included in `infra/aws/queue-sso.json`.
The controller also restores the character after a display Pi reboot, even when
the media source is temporarily unavailable.

When AWS relay sign-in expires, dashboard health displays the profile-specific
renewal command (`aws sso login` for Identity Center, or `aws login` for the
source login profile). The consumer retries automatically after renewal; network
and permission failures remain distinct from expired-login guidance.

The Pi display disables X11 screen blanking and DPMS after opening its desktop
connection, avoiding startup authentication races and the default ten-minute
black screen. Set `display_keep_awake: false` in the device settings to retain
the desktop power-saving behavior.

Spotify control can be auto-approved explicitly by adding `control_spotify` to
`approvals.allowed_tools` while keeping `spotify.playback_control` enabled. It
is not in the default approval list. Playback targets the configured device name
and reports an unavailable target instead of using another active speaker.
Streaming speech generates ahead in a bounded queue and combines PCM into
one-second packets to reduce small-request overhead on a shared Pi.

Voice schedule questions use the read-only `get_calendar_agenda` MCP tool for today or tomorrow, backed by the existing calendar cache. It reports freshness and includes upcoming timed events only; no additional calendar plugin is needed.

The voice session also exposes `get_current_media`, `get_device_health`,
`get_recent_notifications`, `get_focus_session`, and `control_focus_session`.
These share the dashboard controller and durable inbox rather than starting extra
pollers. Calendar queries accept arbitrary ISO `start_date` / exclusive `end_date`
ranges (up to 366 days per query, 200 returned events) including all-day events
and recurrences. The today/tomorrow shorthand remains a fast upcoming-timed-event
cache. Feed/event/notification text is data, never instructions.

Dashboard and MCP share `Application.execute` for volume, focus, notification
preferences/dismissal, reminder scheduling/cancellation, agenda cards, Spotify
Connect discovery, and explicit Echo speech. Voice tools include
`set_speaker_volume`, `snooze_notifications`, `set_notification_source`,
`dismiss_notification`, `list_spotify_devices`, `create_reminder`, `list_reminders`,
`cancel_reminder`, `show_calendar_agenda`, `list_echo_speakers`, and `speak_on_echo`.
Reminders default to a chime and reuse the durable inbox and delivery preferences;
a repeated request ID does not schedule another reminder. Developer controls expose
calendar ranges, Spotify discovery and selected Echo speech. Echo setup is optional:
see [Alexa](docs/alexa.md). No speaker is chosen automatically.