Auto Phone

English | 简体中文

Auto Phone is a command-line tool that controls an Android phone over adb and outputs JSON, along with an MCP server and an agent skill.

This project is developed with vibe coding.

Requirements

• Python 3.10 or later.

• adb installed and on your PATH; PHONE_ADB can point at it instead.

• USB debugging enabled on the phone, authorized after connecting to the computer.

• Installing ADBKeyboard on the phone is recommended.

Related MCP server: adb-mcp-server

Install

Install straight from GitHub (this needs git on your machine). The CLI and the MCP server come from the same Python package:

# pipx (recommended)
pipx install "auto-phone[mcp] @ git+https://github.com/Rynque/auto-phone"

# uv users
uv tool install "auto-phone[mcp] @ git+https://github.com/Rynque/auto-phone"

# or pip
pip install "auto-phone[mcp] @ git+https://github.com/Rynque/auto-phone"

[mcp] is the recommended install: it includes the MCP server dependency and Pillow. Pillow handles screenshot downscaling. If you only use the command line and don't need downscaling, drop [mcp].

After installing, run the self-check:

phone doctor

It checks, item by item: the adb executable, device connection, SDK version, screen state, foreground app, ADBKeyboard, Pillow and the model endpoint. The model endpoint is tested only when PHONE_BASE_URL is set. Anything missing comes with a way to deal with it.

Quick start

phone screenshot shot.png
phone tap 540 1200

The first command saves a screenshot to shot.png. Open the image, find the target, and pass its pixel coordinates to phone tap. Screenshots are full resolution by default, so pixels in the image are screen coordinates and can be used as arguments directly. If you pass --max-width when capturing, convert using orig_width and width from the returned JSON.

Take another screenshot after an action to confirm the result. If the page is still loading or an animation hasn't finished, wait first:

phone wait 1.5

Output format

Except for phone run and phone resume, subcommands print one line of JSON. On success:

{"ok": true, "data": {"tapped": [540, 1200]}}

On failure:

{"ok": false, "error": {"code": "DEVICE_NOT_FOUND", "message": "No online device", "hint": "Check the USB connection and developer options"}}

Failures exit with code 1. hint says what to do next and is also written to stderr. phone run and phone resume print human-readable progress; with --json-events that becomes a JSONL event stream.

Agent integration

Agent skill

npx skills add Rynque/auto-phone

A skill only provides usage instructions — the package above has to be installed first.

MCP server

Register phone-mcp as a stdio server in your MCP host:

{
  "mcpServers": {
    "auto-phone": {
      "command": "phone-mcp"
    }
  }
}

If the command isn't on your PATH, the host can run python -m auto_phone.mcp_server instead.

MCP tool names match the CLI subcommands, plus a sequence tool for running several actions in a row. The model loop behind phone run is not exposed over MCP; the host's own model decides the actions. Coordinates refer to the image returned by the latest screenshot, and the server converts them to screen coordinates automatically.

phone run

phone run starts the built-in vision-model loop: take a screenshot, decide the next action, execute it, verify with another screenshot. It needs an OpenAI-compatible vision endpoint:

export PHONE_BASE_URL=https://api.example.com/v1
export PHONE_API_KEY=sk-...
export PHONE_MODEL=your-vision-model

phone run "open Settings and set the screen timeout to 10 minutes"

When a session is interrupted, its context is saved to .phone-session.json in the current directory; phone resume continues it.

Common options:

• --json-events switches the output to a JSONL event stream, easier for programs to consume.

• --point-confirm draws the coordinates onto the screenshot for confirmation before executing; more accurate, at the cost of one extra model call per step.

• --max-steps caps the number of steps, 50 by default.

Commands

phone screenshot [path] [--max-width N]
phone ui
phone current
phone size
phone devices

phone tap X Y
phone double-tap X Y
phone long-press X Y
phone swipe X1 Y1 X2 Y2 [--duration-ms 300]
phone drag X1 Y1 X2 Y2
phone slide X1 Y1 X2 Y2
phone key back [name ...]
phone type "text"
phone clear-text

phone launch alias-or-package
phone stop alias-or-package
phone clear-bg

phone wait 1.5
phone run "task"
phone resume [instruction]
phone doctor

In detail:

• ui prints the list of interface elements with their center coordinates; use it as a fallback when screenshots fail.

• current prints the foreground app and screen state.

• swipe is a straight-line swipe for scrolling and page turns; --duration-ms defaults to 300.

• drag is the same move at a lower speed, for reordering and sliders.

• slide is for CAPTCHA sliders only; its path includes overshoot and jitter.

• key accepts several key names at once and presses them in order. Names include home, back, enter, del, tab, space, vol-up, vol-down, power, menu, esc, a single letter, a single digit, and numeric keycodes.

• launch and stop accept package names as well as built-in aliases. Aliases work in either language, e.g. 微信 and wechat, 淘宝 and taobao, 抖音 and douyin, 设置 and settings.

Environment variables

• PHONE_DEVICE sets the device serial. It is required when more than one device is connected; otherwise commands fail with AMBIGUOUS_DEVICE.

• PHONE_ADB sets the path to the adb executable. PATH is searched first, and common SDK install locations are probed if that fails.

• PHONE_BASE_URL, PHONE_API_KEY and PHONE_MODEL are the model endpoint, key and model name that phone run needs.

• PHONE_ADBKEYBOARD_APK points at the ADBKeyboard apk file; once set, it is installed automatically when needed.

• PHONE_MAX_STEPS overrides the step cap for phone run, 50 by default.

License

Released under the MIT license.