# mcp-scrcpy-vision
An **MCP server** that gives AI agents complete vision and control over Android devices.
**Features:**
- **Real-time Vision**: Continuous screen streaming via scrcpy H.264 + ffmpeg
- **Fast Input Control**: When streaming, input uses scrcpy control protocol (~5-10ms latency vs ~100-300ms with adb shell)
- **UI Automation**: Element detection via uiautomator with tap coordinates
- **Full Input Control**: Tap, swipe, long press, pinch, drag-drop, text, keycodes
- **System Access**: Shell commands, file transfer, clipboard, notifications
- **Multi-device**: Control multiple Android devices simultaneously
- **WiFi ADB**: Connect wirelessly for untethered automation
---
## Quick Start
### 1. Prerequisites
**Required:**
- Node.js 18+
- ADB (Android Platform Tools) in PATH
- Android device with USB debugging enabled
**For streaming (recommended for fast input):**
- [scrcpy](https://github.com/Genymobile/scrcpy/releases) - download release, extract `scrcpy-server` file
- [ffmpeg](https://ffmpeg.org/download.html) - install and add to PATH
### 2. Install
```bash
git clone https://github.com/anthropics/mcp-scrcpy-vision.git
cd mcp-scrcpy-vision
npm install
npm run build
```
### 3. Configure
Create `.env` file:
```bash
# Required for streaming + fast input
SCRCPY_SERVER_PATH="C:\scrcpy-win64-v3.2\scrcpy-server"
SCRCPY_SERVER_VERSION="3.2"
# Optional (defaults shown)
ADB_PATH="adb"
FFMPEG_PATH="ffmpeg"
DEFAULT_MAX_SIZE="1024"
DEFAULT_MAX_FPS="30"
DEFAULT_FRAME_FPS="2"
```
### 4. Add to MCP Client
**Claude Desktop** (`%APPDATA%\Claude\claude_desktop_config.json` on Windows):
```json
{
"mcpServers": {
"android": {
"command": "node",
"args": ["C:/path/to/mcp-scrcpy-vision/dist/index.js"],
"env": {
"SCRCPY_SERVER_PATH": "C:/scrcpy/scrcpy-server",
"SCRCPY_SERVER_VERSION": "3.2"
}
}
}
}
```
**Cursor** (Settings > MCP):
```json
{
"android": {
"command": "node",
"args": ["C:/path/to/mcp-scrcpy-vision/dist/index.js"],
"env": {
"SCRCPY_SERVER_PATH": "C:/scrcpy/scrcpy-server",
"SCRCPY_SERVER_VERSION": "3.2"
}
}
}
```
### 5. Connect Device
1. Enable USB debugging on Android device (Settings > Developer Options > USB Debugging)
2. Connect via USB
3. Accept RSA fingerprint prompt on device
4. Verify: `adb devices` should show your device
---
## How It Works
### Two Modes of Operation
**1. Snapshot Mode (No streaming required)**
- Uses `android.vision.snapshot` for screenshots
- Input uses ADB shell commands (~100-300ms per action)
- Works without scrcpy/ffmpeg
- Best for simple automation or when streaming isn't available
**2. Streaming Mode (Recommended)**
- Start with `android.vision.startStream`
- Continuous JPEG frames available via resource URI
- Input uses scrcpy control protocol (~5-10ms per action)
- **10-20x faster** than snapshot mode
- Best for real-time control and rapid interactions
### Performance Comparison
| Operation | Snapshot Mode | Streaming Mode |
|-----------|---------------|----------------|
| Tap | ~100-300ms | ~5-10ms |
| Swipe | ~300-500ms | ~50-100ms |
| Type text | ~50ms/char | ~5ms total |
| Screenshot | ~500ms | ~33ms (30fps) |
---
## Tools Reference (32 tools)
### Device Management
| Tool | Parameters | Description |
|------|------------|-------------|
| `android.devices.list` | - | List connected devices |
| `android.devices.info` | `serial` | Get device info (model, SDK, etc.) |
| `android.adb.enableTcpip` | `serial`, `port?` | Enable WiFi debugging |
| `android.adb.getDeviceIp` | `serial` | Get device WiFi IP |
| `android.adb.connectWifi` | `ipAddress`, `port?` | Connect via WiFi |
| `android.adb.disconnectWifi` | `ipAddress?` | Disconnect WiFi |
### Vision
| Tool | Parameters | Description |
|------|------------|-------------|
| `android.vision.startStream` | `serial`, `maxSize?`, `maxFps?`, `frameFps?` | Start continuous stream (enables fast input) |
| `android.vision.stopStream` | `serial` | Stop stream |
| `android.vision.snapshot` | `serial` | Take PNG screenshot (works without streaming) |
| `android.ui.dump` | `serial` | Get UI hierarchy XML |
| `android.ui.findElement` | `serial`, `text?`, `resourceId?`, `className?`, `contentDesc?` | Find elements with tap coords |
### Input Control
**Note:** These automatically use fast scrcpy control when streaming, otherwise fall back to ADB.
| Tool | Parameters | Description |
|------|------------|-------------|
| `android.input.tap` | `serial`, `x`, `y` | Tap at coordinates |
| `android.input.swipe` | `serial`, `x1`, `y1`, `x2`, `y2`, `durationMs?` | Swipe gesture |
| `android.input.longPress` | `serial`, `x`, `y`, `durationMs?` | Long press |
| `android.input.pinch` | `serial`, `centerX`, `centerY`, `startDistance`, `endDistance`, `durationMs?` | Pinch zoom |
| `android.input.dragDrop` | `serial`, `startX`, `startY`, `endX`, `endY`, `durationMs?` | Drag and drop |
| `android.input.text` | `serial`, `text` | Type text |
| `android.input.keyevent` | `serial`, `keycode` | Send keycode |
### App Control
| Tool | Parameters | Description |
|------|------------|-------------|
| `android.app.start` | `serial`, `packageName`, `activity?` | Launch app |
| `android.app.stop` | `serial`, `packageName` | Force-stop app |
| `android.apps.list` | `serial`, `system?` | List installed apps |
| `android.activity.current` | `serial` | Get foreground activity |
### System
| Tool | Parameters | Description |
|------|------------|-------------|
| `android.shell.exec` | `serial`, `command` | Execute shell command |
| `android.file.push` | `serial`, `localPath`, `remotePath` | Push file to device |
| `android.file.pull` | `serial`, `remotePath`, `localPath` | Pull file from device |
| `android.file.list` | `serial`, `path` | List directory |
| `android.clipboard.get` | `serial` | Get clipboard |
| `android.clipboard.set` | `serial`, `text` | Set clipboard |
| `android.notifications.get` | `serial` | Get notifications |
### Screen Control
| Tool | Parameters | Description |
|------|------------|-------------|
| `android.screen.wake` | `serial` | Wake screen |
| `android.screen.sleep` | `serial` | Sleep screen |
| `android.screen.isOn` | `serial` | Check if screen is on |
| `android.screen.unlock` | `serial` | Unlock (unsecured only) |
---
## Resources
The server exposes these MCP resources:
- `android://devices` - JSON list of connected devices
- `android://device/<serial>/frame/latest.jpg` - Latest JPEG frame (when streaming)
---
## Usage Examples
### Basic Automation Loop (Streaming Mode)
```
1. Start stream: android.vision.startStream { serial: "ABC123" }
2. Read resource: android://device/ABC123/frame/latest.jpg
3. AI analyzes image, decides to tap "Login" button
4. Find element: android.ui.findElement { serial: "ABC123", text: "Login" }
5. Tap at returned coordinates: android.input.tap { serial: "ABC123", x: 540, y: 1200 }
6. Wait 500ms, read resource again, repeat
7. When done: android.vision.stopStream { serial: "ABC123" }
```
### Simple Screenshot Mode
```
1. Take screenshot: android.vision.snapshot { serial: "ABC123" }
2. AI analyzes image
3. Find and tap: android.ui.findElement + android.input.tap
4. Take another screenshot to verify
```
### WiFi Connection Workflow
```
1. Connect device via USB
2. android.adb.enableTcpip { serial: "ABC123" }
3. android.adb.getDeviceIp { serial: "ABC123" } → "192.168.1.50"
4. Disconnect USB cable
5. android.adb.connectWifi { ipAddress: "192.168.1.50" }
6. Now use "192.168.1.50:5555" as serial for all commands
```
### App Testing Example
```
1. android.app.start { serial: "ABC123", packageName: "com.example.app" }
2. android.vision.startStream { serial: "ABC123" }
3. Wait for app to load, read frame
4. android.ui.findElement { serial: "ABC123", resourceId: "username_field" }
5. android.input.tap { serial: "ABC123", x: 540, y: 300 }
6. android.input.text { serial: "ABC123", text: "testuser@example.com" }
7. android.input.keyevent { serial: "ABC123", keycode: 66 } // Enter
8. Read frame, verify login succeeded
9. android.vision.stopStream { serial: "ABC123" }
```
---
## Common Keycodes
| Key | Code | Key | Code |
|-----|------|-----|------|
| HOME | 3 | BACK | 4 |
| VOLUME_UP | 24 | VOLUME_DOWN | 25 |
| POWER | 26 | ENTER | 66 |
| DELETE | 67 | TAB | 61 |
| MENU | 82 | APP_SWITCH | 187 |
| WAKEUP | 224 | SLEEP | 223 |
---
## Troubleshooting
### No devices found
```bash
adb kill-server
adb start-server
adb devices
```
Ensure USB debugging is enabled and RSA fingerprint accepted.
### Scrcpy version mismatch
`SCRCPY_SERVER_VERSION` must exactly match your scrcpy-server file. Check the scrcpy release version you downloaded.
### ffmpeg not found
- **Windows**: Download from https://ffmpeg.org/download.html, extract, add bin folder to PATH
- **macOS**: `brew install ffmpeg`
- **Linux**: `apt install ffmpeg` or `yum install ffmpeg`
Or set `FFMPEG_PATH` in .env to the full path.
### uiautomator dump fails
Some devices need screen on. Try `android.screen.wake` first.
### Clipboard not working (Android 10+)
Android 10+ restricts clipboard access. Use UI automation to paste instead.
### Stream won't start
1. Check scrcpy-server path is correct
2. Verify version numbers match
3. Try running scrcpy standalone first to verify it works
---
## Notes & Limitations
- **Fast input when streaming**: When a stream is active, tap/swipe/text/keyevent use the scrcpy control protocol (~5-10ms latency). Without streaming, falls back to `adb shell input` (~100-300ms).
- **One stream per device** at a time
- **Snapshot works without scrcpy** - useful fallback when streaming is not needed
- **Clipboard** has platform limitations on Android 10+
- **Notifications** may require permissions on newer Android
- **Pinch gesture** currently simulates single-finger; true multi-touch requires the streaming session
---
## Security Warning
This MCP server provides **full control** over connected Android devices:
- Execute arbitrary shell commands
- Read/write files on device
- Control UI and input
- Access clipboard and notifications
**Only connect devices you own and trust the AI agent.**
---
## Development
```bash
npm run dev # Development with tsx
npm run build # Compile TypeScript
npm start # Run production build
```
See [claude.md](claude.md) for developer documentation.
See [agents.md](agents.md) for AI agent integration guide.
---
## License
MIT
