Phone-Eye 📱👁️

English | 简体中文

Your AI agent can finally see and operate a real Android phone.

phone_look("what's on screen, where is the login button?")
  → "Login button at (540, 1830) — a green 'Sign in' …"
phone_tap(540, 1830)
phone_look("did the next page load?")

Works with any MCP client — Claude Code, Codex, Cursor, dsh, and friends.

What you need (plain words)

That's everything. No app to install on the phone. No root. No extra server.

Related MCP server: scrcpy-mcp

What it can and can't do (honest table)

The 60-second rule: every Android requires one human moment — enable debugging + authorize once. After that, the phone belongs to your agent, even over Wi-Fi, even after reboots of the computer.

Setup (3 steps)

1. Enable USB debugging (once per phone)

Settings → About phone → tap Build number 7× (unlocks Developer options) → Developer options → USB debugging ON. (Got stuck? Tell us your phone model in Discussions — we'll walk you through it. MIUI/HyperOS may also ask you to sign into a Xiaomi account first.)

2. Install adb (if you don't have it), plug USB once, then go wireless

# macOS: brew install android-platform-tools · Ubuntu/Debian: sudo apt install adb
# Windows: scoop install adb  (or download Android platform-tools)
adb devices          # phone shows up? tap "Allow" on its popup — check "always allow"
adb shell ip route   # ← note the phone's Wi-Fi IP (e.g. 192.168.1.23) while still plugged
adb tcpip 5555       # switch to Wi-Fi mode (adb restarts; the USB entry disappears — normal)
adb connect 192.168.1.23:5555   # use the IP from above; then unplug the cable, forever

Settings → Wi-Fi → your network → details shows the IP; or adb shell ip addr show wlan0 | grep inet.

3. Start phone-eye

git clone https://github.com/boheastill/phone-eye && cd phone-eye
pip install -r requirements.txt

# your eyes — pick ONE:
export PHONE_EYE_VISION_API_KEY=<key>                    # OpenAI / GLM / any compatible
#   (optional: PHONE_EYE_VISION_BASE_URL, PHONE_EYE_VISION_MODEL)
#   local & offline: ..._API_KEY=sk-noauth ..._BASE_URL=http://<host>:8080/v1 ..._MODEL=<your qwen-vl>

python server.py       # stdio MCP server — wire into your client:

Wire it into your client — pick yours:

# Claude Code (easiest):
claude mcp add phone-eye -- python /path/to/phone-eye/server.py
# Codex:
codex mcp add phone-eye --url stdio://python /path/to/phone-eye/server.py  # or see codex docs

// any MCP client (generic stdio shape):
{ "mcpServers": { "phone-eye": { "command": "python", "args": ["/path/to/phone-eye/server.py"] } } }

Docker (optional — no Python needed on the host)

The repo ships a Dockerfile (Python 3.12 + adb):

podman build -t phone-eye .        # or: docker build -t phone-eye .
# smoke: a JSON-RPC initialize reply on stdout means it boots:
printf '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"smoke","version":"0"}}}\n' \
  | podman run --rm -i phone-eye

Use --network host so adb reaches a Wi-Fi phone and your vision endpoint:

"phone-eye": { "command": "podman", "args": ["run","--rm","-i","--network","host",
  "-e","ANDROID_SERIAL=192.168.1.23:5555","-e","PHONE_EYE_VISION_API_KEY=<key>","phone-eye"] }

Recommended vision models (any vision-capable chat model works): gpt-4o-mini (default), GLM glm-4.6v-flash (cheap), or run a local Qwen-VL via llama.cpp for fully-offline — screenshots then never leave your LAN.

What happens when something breaks

• "No Android device reachable" → the tool already tried reconnecting; run adb connect <ip>:5555, or replug USB.

• "No vision server reachable" → you haven't set a key; the error message tells you the exact two fixes.

• Phone rebooted → Wi-Fi adb survives phone reboots on most ROMs; if not, one adb connect again.

• Still stuck? Open a discussion — we answer, and we'll debug your setup with you. Bug reports and "it works on my X" notes are equally welcome.

Tools

Examples

• Mobile web QA loop — the agent verifies its own work on a real screen

• Surviving an OEM setup wizard — vision handles whatever pops up

• Form regression check

• Unattended sentinel — your agent on night watch: wake → unlock → intent → look → screenshot

Curious how it works — or want to modify it? Read the whitepaper (architecture, failure-classification decision tree, security model, how to add a verb). Running it as an always-on HTTP service behind your own fleet? docs/fleet.md.

Why "see", isn't this just adb?

UI-tree-only tools are blind to game canvases, images, and anything the accessibility tree can't show. Vision-only tools drift on coordinates. phone_look fuses both: the model answers what is this, the UI tree supplies exactly where. The day we dogfooded it, it discovered a USB-debugging popup on its own screen, read the buttons, and tapped "Allow" by itself — the story.

License

MIT. Verified on Redmi K40 Gaming / Android 13 — add your device to the table via PR.

Author: Bohea — independent industrial software engineer (Shenzhen). Operator HMIs · device integration · machine data into your customer's ERP. More runnable demos & engineering notes on the site.