J.A.R.V.I.S. (local edition)

Based on adewaskar/jarvis by Aditya Dewaskar, used under the MIT License. All the original work (the holographic interface, the reactor, the voice loop, the Claude Code bridge and its tool gate) is his. This edition changes where the voice runs, what the brain costs, and what the assistant is allowed to touch. For everything not described here, see the original README.

To pull future updates from the original project:

git fetch upstream
git merge upstream/main

What's different

Free local voice. No ElevenLabs and no cloud speech. A small Python service (voice/server.py) runs on your Mac:

• speech to text with faster-whisper

• speaking with Kokoro, with Piper as a fallback

• the wake word with openWakeWord and its "hey jarvis" model

In wake mode a clip that doesn't contain the wake word is dropped before Whisper ever runs. If the service is down, the page falls back to the browser's own voice.

Cost routing. Every request goes to one of two brains, and every answer is logged to logs/brain.log with the brain and model that produced it.

• Local: simple lookups ("what's on my calendar tomorrow", "any new mail from Sam", "summarize my unread mail") go to a local model through Ollama. Free and private.

• Claude: drafting, anything about meetings, calls, notes or follow ups, and anything needing judgment or tone goes to Claude on your Claude subscription.

• Never an API key. The bridge refuses to start if it can see ANTHROPIC_API_KEY (or an auth token or base URL override), and the launcher removes them for Jarvis only, so other tools in your shell keep using them.

• Easy to change. The rules live in bridge/routing.json, a list of patterns per brain, and are read again on every request.

A separate Ollama just for Jarvis. It runs on port 11435 with its models in ./ollama-models inside this checkout, and gets its own home folder (./ollama-home), so it never touches your main Ollama on 11434 or its models. npm start starts it and stops it.

Thunderbird and Granola, with a restricted allowlist. Jarvis reads mail, calendar and meeting notes through the Thunderbird MCP extension and Granola's official MCP server.

• Only named tools are allowed:

  • Thunderbird: list accounts and folders, search and read messages, list calendars and events, save a draft, and open a reply for review.

  • Granola: list meetings, read notes, read a transcript, and search.

• Everything else is denied, including tools the servers add later.

• The local model gets the read tools only.

• Account connectors from claude.ai (your Gmail, Drive and so on) are kept out of Jarvis entirely.

Draft only, never send.

• Jarvis can save a new draft, or open a threaded reply in Thunderbird's compose window for you to review. It cannot send, forward, delete, move or change filters.

• Ask it to send and the bridge itself answers: "I can't send email, but I can save a draft for you to review."

• Replies go to the sender only unless you say "reply all".

• A request that names a person or subject gets a reply to the newest matching email. One that doesn't makes Jarvis ask which email you mean.

Meeting follow ups. "Draft a follow up to Sam from this afternoon's meeting":

• Jarvis finds the meeting in Granola, and asks if more than one matches.

• It reads the notes first, and fetches the transcript only if the notes lack decisions or action items.

• It addresses the attendees other than you, replying in their existing thread if there is one, otherwise writing a new draft from the account that matches their domain (set in .env).

A matching desktop theme. desktop-theme/ makes the rest of macOS look like the HUD: a dim grid and particle wallpaper, desktop widgets for time, meetings, unread mail, weather and system load, a cyan glow on the focused window, and a menu bar with a Jarvis status dot. It is optional, uses about 4.5% of one CPU core, and desktop-theme/restore.sh puts the default look back. See its README.

A macOS desktop app. desktop/ wraps the interface in a frameless Electron window:

• Backdrop: Clear by default (no panel; text keeps a dark halo so it reads over any wallpaper), with a native frosted glass panel and a dark panel as alternatives in the menu bar icon.

• Clicks: in Clear, clicks pass through empty areas.

• Controls: drag by the reactor, Cmd+Shift+J to show or hide, and always on top from the menu.

• Security: it loads nothing but localhost, grants only the microphone, and starts and stops all of Jarvis's services itself.

Standby. Say "standby", "go to sleep", "stop listening" or "that's all" (alone, or as "Hey Jarvis, standby") and Jarvis says "Standing by" and goes back to waiting for the wake word, so nothing else in the room is taken as a request until you say "Hey Jarvis" again. Only whole commands count: "how much sleep did I get?" is still a question.

Meeting mode. A timer over an agenda, for calls. You start it yourself: say "Hey Jarvis, start meeting mode" or press ⌃⌥⌘M. It never starts from the calendar on its own.

• Agenda. If a calendar event is happening now and its description holds a normal agenda, Jarvis offers it. An item is any line with a duration, in parentheses ("Welcome (10 min)", "(1 hr)") or after a separator ("II. PRICING | 20 MINS", "Pricing - 15 min"), optionally led by a time ("9:10") or a numeral ("2.", "II."); the title is kept without them. Lines without a duration are ignored, an item listed twice (summary and detail) counts once, and time left over in the meeting is added as "Buffer". An attached file is not read. Without a timed agenda he asks for items and minutes, or takes them with the command: "start meeting mode, agenda: pricing 15, timeline 10, next steps 5".

• On screen. The current item's countdown sits in the middle of the reactor, with the current item, the next item and the total time left below the rings. The ring pulses amber with two minutes left on an item and red at zero, counting the overrun.

• Private by design. While it runs the microphone is closed completely: no wake word, no recording, no transcription. It is time based only.

• Never into the call. A nudge is a short tone, and only when the output device is headphones by name (add yours with JARVIS_HEADPHONE_NAMES); otherwise nudges are visual only. Setup questions are shown rather than spoken when a meeting is already under way on speakers.

• Shortcuts. ⌃⌥⌘N for the next item, ⌃⌥⌘E to end. In the desktop app they work while the call has focus, and are registered only during a meeting. When it ends, the microphone comes back and Jarvis offers to draft the follow up: in headphones he asks and a "yes" answers; on speakers the offer is shown, and "Hey Jarvis, draft the follow up" (within 30 minutes) drafts it with the meeting's title and agenda.

Meeting tile. What Jarvis shows a call as its camera: a 1280x720 page at http://localhost:5173/tile with the reactor, the current agenda item and its time left, a small "J.A.R.V.I.S." label and a status line. It follows meeting mode live through the bridge's /meeting/state, which carries only the agenda and the timer: never a transcript, a meeting title or anything else.

• Camera. OBS Studio (brew install --cask obs) shows the tile through its virtual camera. node scripts/obs-setup.mjs writes a "Jarvis" profile and scene (one browser source, no audio sources at all) without touching any other OBS profile. Start it with open -a OBS --args --profile Jarvis --collection Jarvis --scene Jarvis --startvirtualcam.

• One time. On first launch click Continue in OBS's permissions dialog without granting anything, then allow "OBS Virtual Camera" under System Settings, General, Login Items & Extensions, Camera Extensions. If the camera then shows OBS's placeholder, stop and start the virtual camera once in OBS.

Joining a Zoom call. "Hey Jarvis, join the meeting" (or ⌃⌥⌘J) takes the Zoom link from the calendar event happening now, or starting within 15 minutes, and joins it in its own Chrome window as "J.A.R.V.I.S.": muted, with the meeting tile as its camera. Then meeting mode starts. "Leave the meeting" (or ⌃⌥⌘L, which works while meeting mode has the wake word paused) closes that window, stops the camera and ends meeting mode. It never joins on its own.

• Its own Chrome. A separate Chrome instance with its own data folder (~/Library/Application Support/Jarvis/zoom-chrome, profile "Jarvis"), driven over Chrome's DevTools pipe rather than a network port.

• No microphone, no sound. The Zoom page is offered one camera (OBS Virtual Camera) and no microphone at all; microphone permission is denied, and Chrome runs with its audio output muted. Jarvis never listens to or records a call.

• One time. If Zoom asks guests to sign in, run node scripts/zoom-profile.mjs and sign in to Zoom in that window.

Spoken introduction. When meeting mode starts, Jarvis generates (with Kokoro) a short introduction from the agenda: "Good afternoon. I'm Jarvis, Sam's AI assistant. I'm here only to keep time on today's agenda: ... I'm not recording or transcribing this call. I'll stay muted from here...", plus a shorter variant chosen under Introduction in the menu bar. It plays only when you press Cmd+Shift+I (registered only while meeting mode runs): Jarvis unmutes its Zoom window, plays the introduction, mutes again and checks that Zoom shows it muted, and tells you the result. The tile shows INTRO READY and MUTED. Set your name with JARVIS_OWNER_NAME in .env.

• Its voice only. Jarvis's Zoom window has one microphone, BlackHole 2ch (a virtual audio device, brew install --cask blackhole-2ch), and nothing feeds BlackHole except the introduction, played by a small helper that plays one file to one named device. Your microphone and the room never reach the call through Jarvis. Without BlackHole the window has no microphone at all, and if your Mac's output or alert sound is set to BlackHole the introduction refuses to play.

• Muted otherwise. Zoom audio is joined muted, and if Jarvis is ever found unmuted outside the introduction it mutes itself again within two seconds.

Related MCP server: local-apple-mcp

Hardware

Tested on one machine: a MacBook Pro with an M4 Max and 36 GB of memory, on macOS. The local model (qwen3:30b-a3b-instruct-2507, about 19 GB) answers a calendar lookup in about seven seconds there. With less memory, choose a smaller model with JARVIS_LOCAL_MODEL. The voice models add about 1 GB of disk and run on the CPU. The desktop app is built for Apple Silicon only.

Setup

You need macOS on Apple Silicon, Node.js 20 or newer, Python 3.11 or newer, Ollama, Claude Code installed and logged in with a Claude subscription, Thunderbird with the Thunderbird MCP extension, and Granola.

1. Install and check.

   npm install
   npm run setup
   cp .env.example .env      # then edit .env

2. Local voice. Follow voice/README.md to create voice/.venv and download the Whisper, Kokoro, Piper and wake word models.

3. The Jarvis Ollama and its model. Start the instance once by hand to pull the model; after that npm start runs it.

   mkdir -p ollama-home
   HOME=$PWD/ollama-home OLLAMA_HOST=127.0.0.1:11435 \
     OLLAMA_MODELS=$PWD/ollama-models ollama serve &
   OLLAMA_HOST=127.0.0.1:11435 ollama pull qwen3:30b-a3b-instruct-2507-q4_K_M

   Keep this checkout on your internal drive. macOS asks for permission before background processes read an external volume, and an Ollama whose models sit on one will hang when started from the desktop app.

4. Mail, calendar and meeting notes. Add both servers at user scope. Run the Granola login in a normal terminal window; it opens a browser to sign in and cannot finish without an interactive terminal.

   claude mcp add --scope user thunderbird -- node /path/to/thunderbird-mcp/mcp-bridge.cjs
   claude mcp add --transport http --scope user granola https://mcp.granola.ai/mcp
   claude mcp login granola
   claude mcp list

   In the Thunderbird MCP extension's settings, leave Block skipReview on (it is on by default). That is what makes an opened reply wait for you.

5. Run it.

   npm start

   Open http://localhost:5173 in Chrome, click INITIALISE, and say "Hey Jarvis".

6. The desktop app (optional).

   cd desktop
   npm install
   npm run icon
   npm run build             # installs /Applications/Jarvis.app

   Allow the microphone when macOS asks. To open it at login: System Settings, General, Login Items, then add Jarvis.

Three tests

Say these out loud, or send them to the running bridge without a microphone:

node scripts/ask.mjs "What's on my calendar tomorrow?"
node scripts/ask.mjs "Draft a reply to my most recent email"
node scripts/ask.mjs "Send an email to myself"

The first should be answered by the local model. The second goes to Claude, which asks which email you mean, or drafts a reply if you name one. The third is refused by the bridge. Check which brain answered each:

npm run brains

Known limitations

• Your own speakers can't wake it. The microphone uses echo cancellation, so audio played by the Mac itself (including test clips) never reaches the wake word. Test with your voice.

• Replies open a window rather than a draft. Thunderbird MCP's saveDraft cannot thread a reply, so replies in an existing thread open Thunderbird's compose window instead of saving silently. New emails are saved as drafts.

• The local model needs help with dates. It gets date ranges and times converted for it by the bridge. Thunderbird reports times in UTC, and the model reads them aloud wrongly otherwise.

• Claude still counts against your plan. Claude answers draw on your subscription's usage limits, even though nothing is billed per token.

• Some connectors and features are off by design. Account connectors from claude.ai are unavailable to Jarvis. The desktop app denies the camera, so the vision and hand tracking features only work in the browser.

• The desktop app is signed only ad hoc. It is built locally for Apple Silicon and not notarised, so it runs only on the machine that built it.

• It has been tested on one machine, with one set of mail accounts.

Credits and licences

• Jarvis: adewaskar/jarvis by Aditya Dewaskar, MIT. See LICENSE for the original copyright notice; this edition is released under the same licence.

• Fonts, stored in public/fonts/ so the page loads nothing remote:

  • Chakra Petch, copyright 2018 The Chakra Petch Project Authors, SIL Open Font License 1.1 (public/fonts/ChakraPetch-OFL.txt).

  • JetBrains Mono, copyright 2020 The JetBrains Mono Project Authors, SIL Open Font License 1.1 (public/fonts/JetBrainsMono-OFL.txt).

• Voice: faster-whisper, Kokoro (kokoro-onnx), Piper and openWakeWord are installed by voice/README.md and are not part of this repository.