Family Relay - pass it on, by voice

"Tell Hannah to eat dinner, there's kimchi stew in the fridge." Hours later Hannah asks the kitchen display what time it is - and hears "It's 6:05. By the way, Dad asked me to tell you to eat dinner - there's kimchi stew in the fridge." She answers "Tell him it was delicious", and when Dad gets home and asks "Did Hannah eat?" he hears "Yes - she said it was delicious, and she heard your message at 6:05."

Family Relay is an Alexa+ add-on for the shared family device. Nobody enrolls and nobody is ever asked who they are: the device learns the household's voices from everyday use, picks up names from how the family talks ("Hannah, don't forget your lunch!" - "Got it!"), and passes a message on only when the recipient's voice is a clear match - woven into whatever they asked for. Everything fades after 24 hours.

Built for the Amazon Developer Hackathon (Alexa+ track) as a self-hosted MCP server (MCP spec 2025-11-25, Streamable HTTP) with MCP Apps cards, plus a web simulator that is a real MCP client standing in for Alexa+ (the Alexa+ add-on toolkit is partner-only).

Architecture: one module of ours, the rest is AWS

 family member speaks
        |  audio (mic recording or demo clip)
        +-------------------------------+
        v                               v
 +---------------------+     +-------------------------------------+
 | Amazon Transcribe   |     | familyvoice  (OUR MODULE)            |
 | speech -> text      |     | enrollment-free speaker recognition  |
 +---------------------+     | ResNet34-LM voice embeddings,        |
        | text               | self-forming voice clusters,          |
        |  + name cues ----->| names learned from conversation       |
        v                    +-------------------------------------+
 +---------------------+               | confirmed / tentative / unknown (+ name)
 | Amazon Bedrock      |               v
 | Nova 2 Lite         |  tools/call  +-------------------------------+
 | ("Alexa+" brain)    | -----------> | Family Relay MCP server        |
 +---------------------+   JSON-RPC   | 6 tools, 3 ui:// cards,        |
        | reply text                  | 24 h memory, delivery receipts |
        v                             +-------------------------------+
 +---------------------+
 | Amazon Polly        |  -> the assistant speaks
 +---------------------+

Why speaker identification is ours: no AWS service recognizes household speakers across sessions without enrollment (checked 2026-10-22, sources in the familyvoice README): Amazon Connect Voice ID is enrollment-based and reached end of support on 2026-05-20; Amazon Transcribe diarization labels speakers only inside one recording; Chime SDK speaker search needs enrollment; Alexa voice ID needs "Alexa, learn my voice"; Nova 2 Sonic lists no speaker identification. That gap is the reason this module exists.

On-device path (FR_PROFILE=local)

Everything except the LLM runs on the device CPU. Measured on an Arm Graviton CPU limited to 2 cores / 2 threads (Graviton2 Neoverse N1 2.5 GHz unless noted); a real Echo-class Cortex-A55/A53 is roughly 2.6-3.4x slower (same models on a Raspberry Pi 4 vs this CPU).

Related MCP server: Alexa Language Bridge

How it works

• Two ears per utterance. The audio goes to STT (what was said) and to the speaker module through POST /speaker/observe on the MCP server (who said it). The module answers confirmed | tentative | unknown; only confirmed with a learned name lets the relay act for a person. Anything less is silence - the request is answered normally and the next utterance is judged again. Nobody is ever asked "who are you?".

• Names come from conversation. Each transcript goes back to the module (note_transcript): "It's Dad", someone calling "Hannah!" and the voice that answers, "Tell Hannah..." (the speaker is not Hannah). A name binds only with enough evidence from at least two conversations.

• Identity is never the LLM's job. The host names the observation in each tools/call (_meta["dev.familyrelay/observation"]), so tools act for the voice that was heard. A different or unsure voice starts a fresh conversation, so nothing said for someone else can leak through the LLM's context.

• Weave-in delivery. When messages wait for the speaker, the assistant calls get_pending_messages alongside the request and adds "By the way, Dad asked me to tell you ...". Senders get a phone notification when their message is heard (or fades unheard); FR_NOTIFY_WEBHOOK_URL posts the same receipts anywhere.

• 24-hour memory. Messages and replies expire and their text is erased.

MCP tools and cards

leave_message, get_pending_messages, reply_to_message, ask_status, get_family_memory, get_speaker_status, each bound to an MCP Apps card (_meta.ui.resourceUri): ui://family-relay/message-card.html (saved / passed on / reply / "Left -> Heard -> Reply"), family-timeline.html (carousel of live items, fading, timeline, notifications), voice-status.html (voice clusters forming, names, discarded noise). 768x480 base canvas; inline / fullscreen / voice-only. Agent Skill: skill/family-relay/SKILL.md.

Run it

pip install -r requirements.txt -r requirements-voice.txt
pip install "familyvoice[device] @ git+https://github.com/moduvoice/familyvoice"   # the speaker module (ffmpeg on PATH for browser audio)
cp .env.example .env                                  # FR_PROFILE=aws: AWS credentials with Transcribe, Polly, Bedrock
python run_local.py                                   # MCP server :8790, simulator :8791

Open http://127.0.0.1:8791/ and press Play family scenario: 30 steps, four family members, the demo clock moves from 7:00 to the next day. Every step is recorded speech (simulator/static/clips/, made with deploy/make_clips_polly.py) that goes through STT and the speaker module; "overheard" steps are family members talking to each other - the speaker module learns from them, the assistant stays quiet. The right column shows the speaker module at work and every MCP JSON-RPC message, voice call and LLM decision.

• FR_PROFILE=local downloads nothing at run time: fetch the models first (deploy/ec2_dev_setup.sh shows the files) and point FR_LOCAL_MODELS at them.

• FR_PROFILE=gpu: deploy/voice_gpu.sh up|setup|stt|tunnel|down runs VoxCPM2 + Qwen3-ASR on a GPU spot.

• FR_PROFILE=demo: a stand-in speaker module driven by the "Speaking now" switch, browser speech.

Tests

python tests/e2e_screenshots.py --video   # plays the scenario headless, screenshots + simulator-run.mp4
python tests/smoke_mcp.py                 # MCP flow over raw JSON-RPC (FR_PROFILE=demo)

Deploy

deploy/template.yaml (SAM): MCP server + simulator on Lambda Function URLs, DynamoDB state, Bedrock Nova 2 Lite permissions; deploy/deploy.sh builds and deploys (creates billable resources).

Notes

• Demo family voices are Amazon Polly (Matthew, Joanna, Ivy, Kimberly); the assistant uses a different voice.

• "Alexa" and Echo are trademarks of Amazon. This is an independent hackathon entry; the simulator uses its own visual design and no Amazon logos.

• The first prototype (Daily Korean Scene) and the earlier VoxCPM2 demo clips are kept under legacy/.

• Honest limits: Alexa+ is simulated here (the add-on toolkit is partner-only) - a real MCP client and server, not a real Echo. Speaker recognition is tested on read English speech with synthetic noise; children and siblings are harder (see the familyvoice README). Not authentication.

License and third-party components

This repository: MIT (LICENSE). No model weights are included; models download at first use.

AWS services used by the submission profile: Amazon Transcribe, Amazon Bedrock (Nova 2 Lite), Amazon Polly.