Android MCP Server

# Quick Start: Scrcpy Frame Streaming ## TL;DR - Fastest Way to Check Screen ```javascript // 1. Start streaming once at app launch await startScrcpyStream(); // 2. In your decision loop, get frame instantly for (let i = 0; i < 100; i++) { const frame = await getLatestFrame(); // <50ms latency const decision = analyzeScreen(frame); if (decision.click) { await touch(decision.x, decision.y); } } // 3. Stop when done await stopScrcpyStream(); ``` **Result:** ~50ms screen check cycles instead of 500-1500ms --- ## Performance Comparison | Method | Setup Time | Per-Frame Latency | Best For | |--------|-----------|-------------------|----------| | **Scrcpy Streaming** | ~2s | <50ms | Continuous agent loops | | **Scrcpy Single Frame** | ~100ms | 100-300ms | Periodic checks | | **ADB Screencap** | None | 500-1500ms | One-off checks | --- ## When to Use Each ### Use Scrcpy Streaming If: - ✅ Agent needs to check screen many times in a loop - ✅ Sub-100ms latency required for responsive decisions - ✅ Long-running automation session (e.g., 50+ interactions) - ✅ Real-time visual feedback important ### Use Single Frame Capture If: - ✅ Occasional screen checks (< 10 per minute) - ✅ 100-300ms latency acceptable - ✅ Don't want persistent stream overhead - ✅ Simple verification after actions ### Use ADB Screencap If: - ✅ Scrcpy not installed - ✅ One-off screenshots - ✅ PNG quality absolutely required - ✅ Don't care about latency --- ## Installation **Windows:** Auto-downloads v2.4 on first use (or `choco install scrcpy`) **Linux:** ```bash apt install scrcpy # Ubuntu/Debian pacman -S scrcpy # Arch ``` **macOS:** ```bash brew install scrcpy ``` --- ## API Reference ### Start Streaming ``` Tool: android_start_scrcpy_stream Args: { deviceSerial?: string } Returns: Success message Latency: ~2000ms (one-time) ``` ### Get Latest Frame ``` Tool: android_get_latest_frame Args: {} Returns: Base64-encoded H.264 frame or null Latency: <50ms ``` ### Stop Streaming ``` Tool: android_stop_scrcpy_stream Args: {} Returns: Success message Latency: <100ms ``` ### Single Frame ``` Tool: android_capture_frame_scrcpy Args: { outputPath?: string, deviceSerial?: string } Returns: Base64-encoded PNG or file path Latency: 100-300ms ``` --- ## Real-World Example **Automating Android registration form:** ```javascript // Good: Streaming await launchApp("pt.washer"); await startScrcpyStream(); // Check form, fill field, verify for (let step = 0; step < 5; step++) { let frame = await getLatestFrame(); // Find input field const field = detectField(frame, step); await touch(field.x, field.y); // Type and verify instantly await typeText(FORM_DATA[step]); frame = await getLatestFrame(); if (detectError(frame)) { console.error("Form error detected"); break; } } await stopScrcpyStream(); ``` **Same example with single frames:** ```javascript await launchApp("pt.washer"); for (let step = 0; step < 5; step++) { let frame = await captureFrameScrcpy(); // ~200ms per frame const field = detectField(frame, step); await touch(field.x, field.y); await typeText(FORM_DATA[step]); frame = await captureFrameScrcpy(); // ~200ms per frame if (detectError(frame)) break; } ``` **Result:** - Streaming: ~500ms total (50ms × 5 frames + setup) - Single frames: ~2000ms total (200ms × 5 frames) - **4× faster** with streaming --- ## Troubleshooting | Issue | Fix | |-------|-----| | "scrcpy not found" | Install scrcpy or let MCP auto-download on Windows | | getLatestFrame returns null | Call startScrcpyStream first | | High CPU usage | Reduce --max-fps in adb-wrapper.ts | | Connection drops | Check ADB connection: `adb devices` | | Slow frames | Increase --video-bit-rate or check bandwidth | --- ## Configuration Adjust streaming parameters in `src/adb-wrapper.ts`: ```typescript spawn(this.scrcpyPath, [ '--max-fps=30', // 30fps (adjust down to 10-15 for lower CPU) '--video-bit-rate=5M', // 5Mbps (increase for better quality) '--video-codec=h264', // H.264 hardware encoding ]) ``` Common adjustments: - **Lower CPU:** Change `--max-fps=15` - **Better quality:** Change `--video-bit-rate=10M` - **More efficient:** Change `--video-codec=vp9` --- ## Tips 1. **Always stop streaming** when session ends to free resources 2. **Use streaming for loops**, single frame for occasional checks 3. **Preload stream** during app launch for smooth decision-making 4. **Monitor CPU usage** if streaming continuously 5. **Check logs** if frames seem stale (may need to restart stream)