Jarvis LifeOS

A voice-first AI study and productivity assistant, built for the Amazon Developer "Build, Ship, Shape" hackathon (Alexa+ track). It has two surfaces over one set of tools:

1. A self-hosted MCP server at /mcp (MCP spec 2025-11-25, Streamable HTTP). Any MCP client can manage the user's tasks and study plans through it. MCP is the open standard behind Alexa+ integrations.

2. A simulated Alexa+-style web experience. Talk or type to Jarvis. It picks a backend tool, the server validates the arguments and runs the operation against persistent storage, and the dashboard shows the real result: which tool ran, with which arguments, and whether it succeeded, failed or needs your confirmation.

We have not connected this to Alexa+ itself (that requires Amazon's Alexa+ preview onboarding) and it does not run on Alexa devices. The MCP server is real and spec-compliant, and we tested it with the official MCP SDK client. The voice web app is a simulation.

Features

• Conversation with voice and text. Browser speech recognition for input (where supported), speech synthesis for replies, and a text box that always works.

• Real tool calling. createTask, listTasks, updateTask, completeTask, deleteTask, generateStudyPlan, updateStudyPlan and getDailySummary. Each one validates its arguments with zod and changes persisted data. No result is hard-coded.

• Task management. Create, list (pending/completed/all), edit title, due date and priority, complete, and delete, from chat or from the Tasks page.

• Study planner. "Create a study plan for tomorrow with two hours of graph algorithms and one hour of dynamic programming" becomes a timed schedule: focus blocks of at most 60 minutes with breaks between them. The plan is saved, and you can change it ("add 30 minutes of recursion to tomorrow's plan", "remove dynamic programming", "start the plan at 2pm").

• Tool transparency. Each assistant reply lists the tool calls it made, with status, the real result, any error, and expandable arguments. The Activity page shows the full log.

• Confirmation for destructive actions. Deleting a task, or replacing an existing study plan, waits for an explicit Confirm or Cancel.

• Two AI engines:

  • Claude (Anthropic API, tool use) when ANTHROPIC_API_KEY is set.

  • Offline intent engine when no key is set. It is a deterministic rule-based parser that calls the same tools. It understands fewer phrasings, and the UI labels its replies "offline engine". If a Claude call fails, Jarvis falls back to it and shows the API error.

Related MCP server: notion-mcp

MCP server

• Endpoint: POST/GET/DELETE http://localhost:8787/mcp, Streamable HTTP with sessions (Mcp-Session-Id)

• Protocol: negotiates 2025-11-25, the latest version, and falls back to older versions for older clients. Built with @modelcontextprotocol/sdk 1.32.

• Tools: the same eight tools, with JSON Schemas, titles and annotations (readOnlyHint, destructiveHint, idempotentHint). Results include readable content and structuredContent (actionId, status, result).

• Human confirmation: deleteTask, and generateStudyPlan when it would replace a plan, use MCP elicitation to ask the user if the client supports it. Otherwise the action stays pending until someone confirms it in the dashboard.

• Shared state: MCP calls write to the same store and appear in the dashboard's Activity log, tagged "MCP client".

• Security:

  • Host-header and Origin validation protect against DNS rebinding.

  • Set MCP_AUTH_TOKEN to require Authorization: Bearer <token>. Do this before exposing the server beyond localhost, and add your tunnel hostname to MCP_ALLOWED_HOSTS.

Connect a client. Run npm run dev, then point any Streamable HTTP MCP client at http://localhost:8787/mcp. For example, the MCP Inspector:

npx @modelcontextprotocol/inspector

Choose transport "Streamable HTTP" and URL http://localhost:8787/mcp.

Setup and running

1. Requirements

• Node.js 22.9 or newer with npm. The scripts use node --env-file-if-exists and node --test glob patterns, which older versions lack. We tested on Node 25.9 and npm 11.12 on Windows 10.

• Git.

• For voice input: Chrome or Edge. Typing works in every browser.

Check your versions:

node -v

2. Install

From the repository root (an npm workspace with client/ and server/):

npm install

For a clean, reproducible install from the lockfile, use npm ci instead.

3. Configure the environment (optional)

The app runs with no configuration: it uses the offline intent engine and stores data in server/data/db.json. To enable Claude, create .env in the repository root.

macOS/Linux/Git Bash:

cp .env.example .env

Windows PowerShell:

Copy-Item .env.example .env

Then edit .env and set ANTHROPIC_API_KEY=.... The server reads .env at startup, so restart it after editing. The dashboard footer and GET /api/health show the active engine.

4. Start (development)

npm run dev

This starts both processes:

• Web app: http://localhost:5173. Vite proxies /api to the API.

• API and MCP server: http://localhost:8787. MCP endpoint: http://localhost:8787/mcp

5. Start (production build)

npm run build

npm start

This serves the built web app, the API and /mcp together on http://localhost:8787.

6. Test

npm test

Runs 29 tests: REST, the assistant scenarios, the Claude agent loop (against a local mock of the Messages API) and the MCP protocol. Each test uses a temporary data file and its own server on a random port, so it never touches your data and doesn't need a running server, network access or an API key.

7. Check the MCP endpoint by hand

With the server running:

curl -i http://localhost:8787/mcp -H "content-type: application/json" -H "accept: application/json, text/event-stream" -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"curl","version":"1"}}}'

The response carries an mcp-session-id header and "protocolVersion":"2025-11-25". Send that header (and mcp-protocol-version: 2025-11-25) on later requests such as tools/list and tools/call.

Environment variables

Architecture

browser (React + Vite)                     server (Node + Express)
┌──────────────────────────┐   /api/chat   ┌─────────────────────────────────────┐
│ Chat + voice orb         │ ────────────▶ │ agent/index.js  picks the engine    │
│  SpeechRecognition (in)  │               │  ├─ agent/claude.js  Claude tool use │
│  speechSynthesis (out)   │               │  └─ agent/local.js   offline parser │
│ Tasks / Study plan views │  REST /api/*  │            │ tool calls              │
│ Activity log + confirm   │ ◀──────────▶  │ executor.js  validate → confirm? →  │
└──────────────────────────┘               │              run → log action        │
 any MCP client ──── /mcp (Streamable HTTP) ▶│ mcp.js  tools/list, tools/call ──┘   │
                                           │ tools.js     8 tools (zod schemas)  │
                                           │ store.js     JSON file, atomic write│
                                           └─────────────────────────────────────┘

• One tool catalogue (server/src/tools.js) serves Claude, the offline engine and the REST endpoints. The JSON Schemas sent to Claude are generated from the same zod schemas the server validates with, so the two can't drift apart.

• The executor (server/src/executor.js) normalizes relative dates, validates arguments, asks for confirmation when a tool requires it, runs the tool, and writes an action record for every attempt, including failures.

• Storage (server/src/store.js) is a JSON file written atomically (temp file, then rename) with serialized writes. We chose it because neither MongoDB nor Firebase was available in the build environment. Everything goes through a small Store class, so it can be swapped for MongoDB or Firestore.

• Dates. The browser sends its local date with each request, so "tomorrow" means the user's tomorrow, not the server's.

API

Mutating endpoints return { action }. An action is { id, tool, args, status: success|error|awaiting_confirmation|cancelled, result, message, error, source, provider, request, createdAt }. Validation failures return 400, unknown tasks return 404, and actions awaiting confirmation return 202.

Testing

npm test starts the real Express app on a random port with a temporary data file and the offline engine. The tests cover:

1. "Create a task to revise binary trees tomorrow": createTask runs, the task is saved with tomorrow's date, and it shows up in /api/tasks

2. "Show my pending tasks": listTasks returns the stored records

3. "Create a study plan for tomorrow with two hours of graph algorithms and one hour of dynamic programming": the plan is scheduled (09:00–12:20, 2h topic split into 2 blocks) and saved

4. "Mark graph revision complete": the right task is completed and leaves the pending list

The tests also cover changing a plan by request, the confirmation and cancel flow, task edits, the daily summary, argument validation (empty title, bad enum, impossible date), ambiguous task references (Jarvis returns the candidates instead of guessing), the action log, persistence across a restart, and generation of the tool JSON Schemas.

The MCP tests (server/test/mcp.test.js) drive /mcp with the official MCP SDK client over Streamable HTTP. They cover:

• initialize negotiating 2025-11-25

• tools/list schemas and annotations

• real tools/call operations that are visible through the REST API and the action log

• tool errors for invalid arguments and unknown tools

• confirmation, both pending without elicitation and declined or accepted through elicitation

• session rules (400 without a session, 404 for an unknown session, DELETE ending a session)

• Host and Origin rejection

• bearer-token auth

The Claude agent tests (server/test/claude-agent.test.js) point the real Anthropic SDK at a local mock of the Messages API (ANTHROPIC_BASE_URL) that returns scripted responses. They check:

• the request we send: model, system prompt with today's date, all 8 tool JSON Schemas, output_config.effort

• the tool loop: tool_use is executed and its tool_result returned, with several tool calls in one turn sharing one user message

• errors and confirmations reported back to the model

• history normalization

• refusal handling and the fallback to the offline engine on API errors

They do not test which tools the real Claude model chooses. That needs a real API key.

All 29 tests pass.

We also ran the four scenarios by hand in the browser, plus confirm/cancel, inline task editing, page reload persistence, and desktop and mobile layouts.

Verification status (2026-10-22, Windows 10, Node 25.9)

Tested and working:

• npm test: all pass, both in the working copy and in a fresh git clone + npm ci (23/23 at that point; 29/29 after the Claude agent tests were added).

• npm run build + npm start from a fresh clone. The built web app, /api and /mcp are all served together.

• /mcp over raw HTTP (curl):

  • initialize returns protocol 2025-11-25 and an mcp-session-id

  • notifications/initialized returns 202

  • tools/list returns all 8 tools

  • tools/call writes real records to the data file, and invalid arguments return isError: true

• MCP persistence across a server restart: a task and plan created over MCP before the restart could be listed, changed and completed after it, and the REST API showed the same records.

• Web app against the production build: the four scenarios, a plan change, and a delete confirmed through the UI, all reflected in the API, with no browser console errors.

• Dev mode: the Vite proxy forwards /api correctly. Cross-site browser origins get 403.

Not verified:

• Real Claude tool selection (blocked: no API key on the build machine). Our side of the integration is covered by the mock-API tests above. With an invalid key, the SDK call reaches the Anthropic API, the 401 is handled, and the offline engine takes over with a visible notice.

• Voice input with a real microphone. Speech output was observed working.

• Third-party MCP clients (MCP Inspector, Claude Desktop) and any connection to Alexa+ itself. MCP was tested with the official @modelcontextprotocol/sdk client and with raw HTTP.

• Node versions below 25.9, macOS and Linux.

Limitations

• On Windows, stopping npm start from a parent process (not a terminal Ctrl+C) can leave the child node process holding port 8787. Stop it from Task Manager, or run node src/index.js in server/ directly.

• The offline engine only understands the phrasings it was written for. Use Claude for open-ended language.

• Speech recognition depends on the browser: Chrome and Edge support it, Firefox does not. Typing always works.

• Single user and no authentication. The JSON file store suits a demo, not a multi-user deployment.

• Task completion dates use the server's local timezone.

Project structure

client/                 React + Vite dashboard
  src/App.jsx           layout, state, data refresh
  src/components/       Chat, ActionCard, Tasks, Plans
  src/useSpeech.js      speech recognition + synthesis hooks
  src/api.js            API client
server/
  src/index.js          entry point
  src/app.js            Express routes
  src/tools.js          tool catalogue + scheduling
  src/executor.js       validation, confirmation, action log
  src/store.js          JSON-file persistence
  src/dates.js          date helpers
  src/agent/            claude.js, local.js, index.js
  src/mcp.js            MCP server (Streamable HTTP, 2025-11-25)
  test/api.test.js      end-to-end REST + agent tests
  test/mcp.test.js      MCP protocol tests
  test/claude-agent.test.js  Claude agent loop vs. a local mock API
docs/SUBMISSION.md      Devpost text, demo script, feedback drafts

Built with

React 19, Vite 7, Node.js, Express 5, zod 4, the Model Context Protocol TypeScript SDK (@modelcontextprotocol/sdk), the Anthropic TypeScript/JavaScript SDK (@anthropic-ai/sdk) with Claude tool use, and the Web Speech API.

License

MIT. See LICENSE.