JetKVM MCP Server

This is a stdio server that opens JetKVM's official local Web UI with Playwright and provides screen capture and HID input of the connected computer as MCP Tools.

In this document, the computer connected to JetKVM and receiving operations is called "PC1", and the computer running the MCP Server and Playwright is called "PC2". HID refers to mouse and keyboard inputs sent by JetKVM to PC1.

Implemented Features

• PNG capture at the same pixel dimensions as the video frames received from PC1

• Absolute mouse movement, click, double-click, scroll

• Single key, macOS hotkey, printable ASCII input

• macOS lock screen detection requiring multiple screen features, with unlock attempts limited to a maximum of one

• Persistent reuse of BrowserContext, WebRTC, and HID DataChannel

• One-time reconnection on WebRTC disconnection, with HTML/PNG diagnostic saving

• Output directory restriction, rejection of file names pointing outside the directory, and suppression of credentials in logs

Related MCP server: Playwright MCP

Approach Based on Investigation

We checked the JetKVM official jetkvm/kvm repository (dev, commit b3c29a44d9e2862b8ff7530830781803ce27b060) as of 2026-08-18.

• The local authentication UI uses POST /auth/login-local and sets an HttpOnly authToken Cookie on success.

• Local WebRTC signaling uses the authentication-protected GET /webrtc/signaling/client.

• The UI adds a recvonly video transceiver to RTCPeerConnection and sets the received MediaStream as the srcObject of a <video> element.

• This implementation runs the official UI as-is with Playwright, draws decoded video frames onto a canvas, and converts them to PNG.

We do not use custom signaling, Developer Mode, custom Firmware, Cloud/Remote Access, or JetKVM configuration changes. Virtual media, Wake on LAN, Terminal, Serial, etc., are also not exposed.

Architecture

When the MCP Server starts, it creates one Playwright Chromium, one BrowserContext, and one page, logs into JetKVM once, and waits until the WebRTC video is ready. All Tools share the same page and WebRTC/DataChannel session; concurrent calls are processed sequentially. Normal Tool calls do not restart the browser or re-login.

We do not use Playwright's page.mouse / page.keyboard for input. These only operate the Chromium on PC2 and cannot guarantee delivery to PC1.

For mouse and keyboard, we preferentially call window.__kvmTestHooks exposed by the JetKVM official Web UI for E2E testing. If the hook is unavailable, we send DOM events to the event listeners registered by the official UI on <video> and document. Scrolling always goes through the official UI's video wheel listener. This design avoids implementing custom HID packets and reuses the official UI's HID RPC handshake, DataChannel selection, and fallback for older versions.

__kvmTestHooks is not a stable external API of JetKVM. Since this implementation verified the implementation at the above commit, please re-validate input compatibility after JetKVM updates.

Main components:

FLOW of Tool calls:

MCP client
  → Zod引数検証
  → JetKvmSession内の直列実行キュー
  → WebRTC video健全性確認
  → 映像取得、または公式UIのHID/RPC経路
  → MCP response

On WebRTC disconnection, reload the same page once and reconnect. If recovery does not occur within 30 seconds, save diagnostic files and return an error including the possibility of another JetKVM WebRTC session.

JetKVM may have conflicts with simultaneous WebRTC sessions. While the MCP Server is in use, do not open the same JetKVM KVM screen in a normal Chrome/Safari, etc.

Official references:

• https://github.com/jetkvm/kvm

• https://github.com/jetkvm/kvm/blob/dev/ui/src/routes/login-local.tsx

• https://github.com/jetkvm/kvm/blob/dev/ui/src/routes/devices.%24id.tsx

• https://github.com/jetkvm/kvm/blob/dev/ui/src/components/WebRTCVideo.tsx

Setup

Node.js 20 or higher is required.

npm install
npx playwright install chromium
export JETKVM_URL=http://jetkvm.local
export JETKVM_PASSWORD='your-local-password'
export JETKVM_SCREENSHOT_DIR=./screenshots
export JETKVM_PC_PASSWORD='your-pc1-macos-password'
npm run build

When using .env, the server itself does not automatically load dotenv, so load it in the startup shell.

cp .env.example .env
# .envへ実値を設定（Gitにはcommitしない）
set -a
source .env
set +a
npm run build
npm start

Only the first time, Chromium is installed. No need to re-run for normal startups that do not update dependencies.

npx playwright install chromium

JETKVM_PC_PASSWORD is exclusively for unlocking PC1 macOS. Do not pass it as a Tool argument; manage it only in the local .env on PC2. .env is gitignored, but do not accidentally duplicate it under a different name. It is recommended not to write it in plain text in configuration files like Hermes, but to inherit environment variables from the startup shell.

Direct Verification of PNG Capture

npm run screenshot -- current-screen.png

On success, save screenshots/current-screen.png. The file name cannot go outside JETKVM_SCREENSHOT_DIR and only .png is allowed.

Each execution waits 5 seconds for SPA initialization, then saves and displays the following diagnostic information to stderr before waiting for video. Diagnostic files remain even if video cannot be obtained.

• Current URL, page title, first 2000 characters of body

• Number of elements: video, password input, form, #root, text=JetKVM

• screenshots/debug-page.html

• screenshots/debug-page.png (full-page)

MCP Configuration Example

{
  "mcpServers": {
    "jetkvm": {
      "command": "node",
      "args": ["/path/to/jetkvm-mcp/dist/server.js"],
      "env": {
        "JETKVM_URL": "http://jetkvm.local",
        "JETKVM_PASSWORD": "<local-password>",
        "JETKVM_SCREENSHOT_DIR": "/path/to/jetkvm-mcp/screenshots"
      }
    }
  }
}

Exposed Tools and MCP arguments:

Screenshot write destination is restricted to screenshots/ directly under the server process's current directory. If JETKVM_SCREENSHOT_DIR is specified, it must match this location after normalization. File names pointing outside this directory, such as ../ or absolute paths, are rejected.

Safety Specification for PC1 Unlock

Lock state is determined by OCR (Tesseract.js WASM with English and Japanese data bundled) on PC2, analyzing PC1 video by region, and returning a ternary value: locked / unlocked / unknown. Images and OCR results are not sent to external services.

OCR implementation: https://github.com/naptha/tesseract.js

• locked: All three types (time, date, password prompt) are confirmed in the specified region.

• unlocked: No password prompt, and three or more known macOS menu bar words are confirmed at the top of the screen.

• unknown: The above evidence is not complete. No password or Enter is sent.

This is not a method to obtain macOS state via OS API, but a conservative judgment based on character arrangement on the screen. It may become unknown due to display language, resolution, wallpaper, or macOS UI changes. To avoid erroneous input, priority is given to not attempting unlock when evidence is insufficient.

unlock_pc() and ensure_unlocked() take no MCP arguments. Credentials are read only from JETKVM_PC_PASSWORD and are not included in logs, exceptions, MCP responses, or file names. Credential input uses a dedicated internal HID path that does not output diagnostic logs. Per Tool call, password input and Enter are performed at most once, with no automatic retry. Judgment images are saved as unlock-before.png, ensure-unlocked-before.png, and result confirmation as unlock-after.png only within screenshots/.

The return value status is one of unlocked, already_unlocked, not_lock_screen, state_unknown, or unlock_failed.

Tests

npm test
npm run build

Roadmap

Future candidates:

• Reduce state judgment latency by reusing OCR worker within the session

• Increase lock judgment fixtures for macOS display language, resolution, and wallpaper variations

• Structured audit events per input Tool (not including secrets)

• Read-only health Tool to check WebRTC/DataChannel status without input

• Startup wrapper for Hermes Agent that does not write secrets directly in configuration files

Explicit non-goals:

• Use of Developer Mode, custom Firmware, Cloud/Remote Access

• Exposure of JetKVM configuration change API, Terminal, Serial, virtual media, Wake on LAN

• Direct string injection into Japanese IME, automatic retry on authentication failure

Real Device Verification Logs

• 2026-08-18 STEP 1: Executed take_screenshot 3 times and move_mouse 2 times in the same WebRTC session.

• (100,100) → HID (1708,3037), (1700,900) → HID (29028,27331).

• Both confirmed official E2E HID hook, HID ready, RPC DataChannel open, WebRTC connected.

• Confirmed that PC1 cursor moved to two different points in mouse-a.png and mouse-b.png.

• Real device calls for click, double_click, scroll, press_key, hotkey, type_text: 0 times.

• 2026-08-18 STEP 2: Executed take_screenshot 2 times, move_mouse 1 time, left click 1 time in the same WebRTC session.

• (960,540) → HID (16392,16399). Both move/click confirmed official E2E HID hook, HID ready, RPC DataChannel open, WebRTC connected.

• Since a safe lock screen background was clicked, no PC1 UI change other than cursor movement.

• Real device calls for double_click, right click, scroll, press_key, hotkey, type_text in STEP 2: 0 times.

• 2026-08-18 STEP 3: Executed take_screenshot 2 times, press_key("Tab") 1 time (down/up each 1 time) in the same WebRTC session.

• Tab via official sendKeypress E2E HID hook (USB HID usage 0x2b). Confirmed HID ready, RPC DataChannel open, WebRTC connected.

• Could not determine clear focus change on lock screen in before/after images. Real device calls for keys other than Tab, click, double_click, scroll, hotkey, type_text in STEP 3: 0 times.

• 2026-08-18 STEP 4: Executed take_screenshot 3 times, type_text("abc") 1 time, press_key("Backspace") 3 times in the same WebRTC session.

• abc and Backspace via official sendKeypress E2E HID hook. Confirmed HID ready, RPC DataChannel open, WebRTC connected for all inputs. Shift 0 times, Enter 0 times due to lowercase input.

• After input, three character markers appeared in the password field, all disappeared after 3 Backspaces. No transition from lock screen or additional operations.

• 2026-08-18 STEP 5: Executed take_screenshot 2 times, move_mouse(1400,700) 1 time, left double_click(1400,700) 1 time in the same WebRTC session.

• Double-click via official sendAbsMouseMove E2E HID hook, sending left button down/up twice each. Confirmed HID ready, RPC DataChannel open, WebRTC connected.

• Performed on an empty area of the lock screen, no screen state change. Single click and other additional inputs: 0 times.

• 2026-08-18 STEP 6: Executed take_screenshot 2 times, move_mouse(1150,540) 1 time to Slack message body area, scroll(0,500) 1 time in the same WebRTC session.

• Scroll sent via official video wheel listener to JetKVM's wheel RPC path, normalized wheel value (0,-5). Confirmed HID ready, RPC DataChannel open, WebRTC connected.

• Confirmed Slack message body moved upward in before/after. Click, double_click, keyboard Tools and other additional inputs: 0 times.

• 2026-08-18 STEP 7 first attempt: hotkey(["SHIFT","TAB"]) stopped before HID dispatch due to normalization error of uppercase TAB. Screenshots 2 times, HID input to PC1 0 times, no screen change.

• Added fix to normalize TAB alias to Tab and unit test. According to safety conditions, no real device retry was performed in this attempt.

• 2026-08-18 STEP 7 retry: Executed take_screenshot 2 times, hotkey(["SHIFT","TAB"]) 1 time in the same WebRTC session.

• Sent via official sendKeypress E2E HID hook in order: ShiftLeft down (0xe1), Tab down (0x2b), Tab up, ShiftLeft up. Confirmed HID ready, RPC DataChannel open, WebRTC connected.

• PC1 changed from wallpaper-only display to lock screen display. Other input Tools and additional real device inputs: 0 times.