serve-droid

A shared browser cockpit and agent control plane for Android. Stream an emulator or physical device, control it from Chrome or Edge, inspect its semantic UI tree and Logcat, or hand the same session to an MCP-compatible coding agent.

Status: v0.1 development release. The public API is versioned, but video transport and device compatibility still need validation on the published support matrix.

npx serve-droid
# → Browser cockpit at a local authenticated URL

One shared session: compact active-device controls stay on the left while the live Android surface owns the primary workspace. On wide screens the agent inspector floats contextually on the right and switches between filtered Logcat, semantic UI, and privacy-safe Activity; on narrower windows it stacks below the device instead of shrinking the interaction surface. The deterministic demo also exercises authenticated file install and clipboard flows. It is not real-device validation; hardware evidence is tracked separately in the release checklist.

What you get

• H.264 device streaming in a responsive browser cockpit with direct pointer DOWN, coalesced MOVE, and UP/CANCEL events carried by the same live scrcpy session before release.

• Mouse-wheel and trackpad scrolling over the device, coalesced into bounded scrcpy touch gestures so high-resolution browser input cannot create a device-command backlog.

• One-click screenshot capture from the decoded live frame, with an authenticated device fallback, preview, download, and supported-browser clipboard actions.

• Exact semantic element targeting that stops on missing or ambiguous matches.

• Searchable, priority-filtered Logcat with pause, clear, and copy controls.

• Explicit Android system volume down/up controls in the device rail, separate from browser playback mute.

• Privacy-filtered in-memory Activity timeline for recent structured session, device, and control events.

• Secure LAN token entry and bounded clipboard paste into the focused Android field.

• Drag and drop APK or file transfer with byte progress and an explicit Android processing phase.

• One bounded observation containing the screen, UI hierarchy, foreground app, device state, and incremental logs.

• The same capabilities through CLI, authenticated HTTP/WebSocket APIs, MCP, and an Agent Skill.

Related MCP server: Android MCP

Why

Android Studio mirrors devices, scrcpy provides excellent native display/control, and Maestro is a strong test automation system. serve-droid focuses on a different loop: a human and an AI agent sharing one observable browser session during development and debugging.

Inspiration

The browser-first cockpit direction was inspired by Evan Bacon's serve-sim. serve-droid is an independent Android-focused implementation built around ADB, scrcpy transport, Logcat, semantic observations, and MCP.

Requirements

• Node.js 22 or newer

• Android SDK Platform Tools with adb on PATH, ANDROID_HOME, or ANDROID_SDK_ROOT

• An Android 8 / API 26+ emulator or device visible in adb devices -l

• A current Chrome, Edge, Safari, or Firefox browser. Chromium uses WebCodecs; Safari and Firefox use the higher-cost TinyH264 software fallback.

Android Platform Tools are never downloaded silently or redistributed.

Quick start

npx serve-droid doctor
npx serve-droid doctor --browser
npx serve-droid avd list
npx serve-droid start --detach
npx serve-droid list --json
npx serve-droid activity --since 0 --json

serve-droid activity reads the same bounded, privacy-filtered session timeline shown by the browser inspector. It selects one live session, authenticates with that session internally, and supports cursor resume without exposing the bearer token. See the Activity guide.

When selecting a fixed port, probe it before touching the device. Occupied ports return the stable PORT_IN_USE code and exit status 31; --port 0 keeps safe ephemeral allocation.

npx serve-droid doctor --port 47321 --json
npx serve-droid start --port 47321

Device playback audio is opt-in and browser playback remains muted until a human enables it:

npx serve-droid start --audio

See device audio and privacy.

Open the printed local URL. The server listens on 127.0.0.1, generates a random token, and injects it into the local UI. To select a device:

npx serve-droid --device emulator-5554

Installed emulators can be managed explicitly with serve-droid avd start <name> and serve-droid avd stop <serial>. No SDK content is downloaded or licensed automatically; see the AVD lifecycle guide.

To inspect several connected devices with independent tokens and bounded resources:

npx serve-droid grid --max-devices 4

See the multi-device grid security and isolation model.

Coordinates are normalized: (0, 0) is the logical top-left and (1, 1) is the bottom-right.

npx serve-droid tap 0.5 0.5
npx serve-droid swipe 0.5 0.8 0.5 0.2 --duration 350
npx serve-droid app deep-link 'servedroid://fixture/example'

Failed ADB interactions preserve generic errors unless Android reports explicit keyguard or input-policy evidence. See the interaction failure diagnostics for the DEVICE_LOCKED, SECURE_SCREEN, and INPUT_RESTRICTED contract.

Direct browser pointers now stream an authenticated begin → move* → end lifecycle over the control WebSocket while the cursor or finger is still moving. The browser keeps one request and one latest move pending, while the server gives the stream exclusive ownership of one scrcpy finger and maps it against a fixed encoded-frame size. If scrcpy is unavailable before begin, the request is rejected before device input and the cockpit safely uses its existing bounded release-time action. After a successful begin, failures are never replayed through ADB. A sparse same-point heartbeat keeps intentional long presses alive, while abandoned streams receive a best-effort cancel after two seconds of inactivity.

In the browser, hover the Android surface and use the mouse wheel or a two-finger trackpad scroll. A burst is coalesced into one bounded, cursor-anchored touch swipe. Modifier-assisted browser zoom gestures are left untouched.

Session capture is explicit and bounded. --record ./recordings stores the original H.264 stream plus privacy-filtered event summaries; it never records tokens, Logcat, typed text, URLs, or file contents. Use --record-controls ./recordings instead to authorize browser start/stop without recording immediately; the cockpit never chooses a host path or limit. See the recording and retention guide. Finalized recordings can also be exported with serve-droid recording trace for timeline inspection in Perfetto.

MCP

{
  "mcpServers": {
    "serve-droid": {
      "command": "npx",
      "args": ["-y", "serve-droid", "mcp"]
    }
  }
}

The MCP surface deliberately provides explicit bounded tools instead of arbitrary shell access. See the MCP guide.

Security

• Loopback-only binding by default.

• Bearer authentication for reads, mutations, video, and control.

• Browser WebSockets carry credentials in a subprotocol header, not a URL.

• No arbitrary ADB or host shell endpoint.

• Uploads are capped at 256 MiB, written to a private temporary directory, and removed immediately.

• clear and uninstall require explicit confirmation from non-interactive clients.

Read SECURITY.md before binding to a LAN interface.

Short-lived remote access through an existing named Cloudflare Tunnel is available only with explicit consent, HTTPS, bearer authentication, and a hard expiry. Read the tunnel threat model and setup guide before using it.

Supported and deferred

v0.1 targets macOS, Linux, Windows, Android API 26+, local emulators, USB devices, Wi-Fi ADB, Chrome, Edge, Safari, and Firefox. Installed AVD lifecycle controls, opt-in bounded local recording, the TinyH264 browser fallback, opt-in device audio for API 30+, and opt-in expiring named-tunnel support are included. Cloud device labs, accounts, multi-user roles, AVD creation/provisioning, iOS, and arbitrary shell access are deferred. See the browser support matrix.

Development

pnpm install
pnpm verify
pnpm --filter @serve-droid/cli dev -- doctor

The Android fixture source is under fixtures/android-test-app. Real-device tests run only when SERVE_DROID_DEVICE_TEST=1 is set.

See the evidence-based release checklist for completed work and the remaining hardware, platform, and publication gates.

Project TODO

• Publish the open repository with protected main, CodeQL, Dependabot, and secret scanning.

• Ship the shared CLI, HTTP/WebSocket, MCP, Agent Skill, and browser cockpit foundation.

• Add a reproducible, clearly labeled cockpit screenshot to this README.

• Refine the browser cockpit into a device-first responsive workspace with contextual inspector.

• Add searchable Logcat controls with priority filtering, pause, clear, and copy.

• Add bounded privacy-safe Activity across browser, CLI, and authenticated HTTP.

• Add secure LAN token handoff and bounded browser clipboard controls.

• Add coalesced mouse-wheel and trackpad scrolling over the Android surface.

• Route browser pointer and gesture input through the active scrcpy control channel.

• Stream direct pointer movement before release with bounded coalescing and cancellation.

• Complete the real-device acceptance matrix on macOS, Linux, and Windows.

• Publish and validate the npm release candidate before tagging v0.1.0.

The complete, evidence-based checklist lives in docs/TODO.md. Items stay unchecked until the repository contains the implementation or the required external evidence.

License

Apache-2.0. See THIRD_PARTY_NOTICES.md.