Skip to main content
Glama
android-phone-mcp-server

android-phone-mcp-server

by expoli

android-phone-mcp

License: MIT CI

代码 Agent 无关的 Android 控制 MCP Server:语义动作 + 验证闭环 + 融合感知。 Any MCP client (Claude Code / Cursor / Cline / 自研 Agent) can control an Android device through semantic tools — no coordinate guessing, every action returns verification evidence.

✅ Phase 0(语义动作 + 验证闭环)/ Phase 1(OCR 融合感知 + 校验工具集 + 多设备并行)/ Phase 2(VLM 视觉融合 + 执行预算 + 增量 diff + 评测)均已完成真机验收。 完整设计见 android-phone-mcp-server-设计文档.md 与 openspec/。

快速开始(开发环境)

# 1. 创建虚拟环境并安装(uv;国内网络请配置镜像)
uv venv .venv
export UV_DEFAULT_INDEX=https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple   # 可选
uv pip install -e ".[dev]"

# 2. 连接 WiFi adb 设备
adb connect <phone-ip>:<port>        # 例如 192.168.1.15:39455
adb devices                          # 确认 device 状态

# 3. 启用开发写权限(默认只读!)
cp .env.example .env

# 4. 启动 MCP server(stdio)
.venv/bin/android-phone-mcp --stdio

# 5. 常用检查
.venv/bin/android-phone-mcp --show-config   # 查看生效配置
.venv/bin/android-phone-mcp --list-tools    # 列出全部 18 个工具

作为 MCP 客户端接入(Python 示例)

import asyncio
from fastmcp import Client
from android_phone_mcp.config import Config
from android_phone_mcp.server import create_server

async def main():
    server = create_server(config=Config(allow_write=True))  # 开发期开写
    async with Client(server) as client:
        r = await client.call_tool("open_settings", {"panel": "about_phone"})
        print(r.data)   # {executed, screen_changed, changed_elements[], screen_hash}

asyncio.run(main())

任意 MCP 客户端(Claude Code / Cursor / Cline / 自研 Agent)均可通过 MCP 协议接入;错误以结构化 JSON 返回(如 WRITE_DISABLED / SELECTOR_AMBIGUOUS + candidates[]),模型可直接读取提示继续操作。

Related MCP server: uiautomator2 MCP Server

安全模型

开关

环境变量

默认

写操作(tap/输入/安装等)

ANDROID_MCP_ALLOW_WRITE

off(只读)

任意 adb shell

ANDROID_MCP_ALLOW_SHELL

off

每动作超时

ANDROID_MCP_ACTION_TIMEOUT

30s

配置优先级:环境变量 > config.yaml > 内置默认。.env 仅用于本地开发一键开写。

工具清单(18 个)

类别

工具

设备

list_devices、get_device_info

观察

get_screen、get_screen_hash、verify_element、diff_state、locate

动作

tap、swipe、scroll_page、type_text、scroll_until、smart_scroll、open_app、open_settings、press_key

等待

wait_for

会话/护栏

reset_session

校验工具(阶段 1 新增):

  • wait_for(target, state=present|absent, timeout=10):阻塞等待元素出现/消失(加载动画)

  • verify_element(target, text_predicate?):断言元素存在/缺失/文本匹配,结构化布尔

  • diff_state():当前 vs 上次快照的增量 diff(added/removed/changed + 双 hash)

  • smart_scroll(target):滚动聚合多屏全部命中(含 OCR 感知),一次调用返回

OCR 融合感知(阶段 1)

无障碍树对 Flutter/Unity/游戏等自渲染引擎失效时,get_screen 自动回退 OCR 文本层:识别文本以 ocr- 前缀伪元素合并进紧凑快照(共享 id 空间),tap/type_text 可直接定位;tap 前会重新 OCR 确认位置(OCR_STALE 兜底)。

# 安装 OCR 引擎(可选 extra,默认不装)
uv pip install -e ".[ocr]"            # 或 pip install android-phone-mcp[ocr]
# 引擎:rapidocr_onnxruntime(py3.14 无 paddlepaddle wheel 时的替代,中文识别佳)

配置

默认

说明

ANDROID_MCP_OCR_ENABLED

true

OCR 兜底开关

ANDROID_MCP_OCR_MIN_INTERACTIVE

3

树可交互元素低于该值触发 OCR

VLM 视觉融合(阶段 2 · 第三感知层)

当无障碍树与 OCR 都无法定位目标时(抽象图形界面/游戏 UI),可选 VLM grounding(OpenAI 兼容视觉端点,如 Qwen2.5-VL via vLLM/Ollama)作为最终兜底:

  • locate(target):按文本/描述返回目标位置 {found, position, box}

  • 自动兜底:语义工具(tap/scroll_until 等)目标未命中且 VLM 可用时,自动 VLM 定位,vlm- 伪元素进快照(共享 id 空间);tap 前重确认(VLM_STALE 兜底)

  • SoM 编号截图通道:vision 模型可直接看编号截图操作

# 安装 VLM 引擎(可选 extra,默认不装)
uv pip install -e ".[vlm]"            # 或 pip install android-phone-mcp[vlm]

配置

默认

说明

ANDROID_MCP_VLM_ENABLED

true

VLM 兜底开关

ANDROID_MCP_VLM_BASE_URL

空

OpenAI 兼容端点(如 http://localhost:8000/v1)

ANDROID_MCP_VLM_MODEL

空

模型名(如 qwen2.5-vl-7b)

ANDROID_MCP_VLM_API_KEY

空

本地端点可留空

VLM 为服务端能力——调用模型不需要视觉;未配置端点时结构化降级(VLM_UNAVAILABLE),不影响树/OCR 路径。

执行预算(阶段 2 · 防失控护栏)

会话级动作数/时长上限,防止 Agent 死循环/狂点:

配置

默认

说明

ANDROID_MCP_BUDGET_MAX_ACTIONS

0(关)

最大写动作数,超出拒绝

ANDROID_MCP_BUDGET_MAX_SECONDS

0(关)

自首个动作起最大时长

超限后写工具返回 EXECUTION_BUDGET_EXCEEDED(含 limit 与恢复提示);调用 reset_session 清除快照缓存并重置预算:

tap(...)                 # ok
tap(...)  # 第 N+1 次   -> {error: "EXECUTION_BUDGET_EXCEEDED", limit: "actions", hint: "..."}
reset_session()          # {ok: true, budget: {action_count: 0, ...}}
tap(...)                 # ok(预算已重置)

AndroidWorld 子集评测(阶段 2)

# 运行评测(需要 WiFi adb 设备)
ANDROID_TEST_DEVICE=<serial> .venv/bin/python -m eval.runner
# 输出 eval/report.json:每任务 {task, success, steps, error/reason}

WiFi adb 注意事项与重连流程

WiFi adb 注意事项与重连流程

  • 调试设备通过 WiFi adb 连接时,请勿在测试中切换 Wi-Fi 开关——关闭瞬间手机会断网,adb 连接随之断开

  • IP:端口会随重连变化;灭屏/省电可能导致连接离线

重连流程:

# 1. 确认离线
adb devices                 # 设备消失或显示 offline

# 2. 断开旧连接并重新 connect(手机端需保持"无线调试"开启)
adb disconnect
adb connect <phone-ip>:<port>

# 3. 验证
adb devices                 # 应显示 device(非 offline)
adb -s <serial> shell getprop ro.product.model   # 输出模型号即正常
  • 若 connect 后反复 offline:检查手机无线调试端口是否变化、与电脑是否同网段、防火墙是否放行 5555 段端口

  • 设备池在每次工具调用前做 health check,连接失效时返回结构化错误,不会卡死调用

验收用例(Phase 0 三用例,均为网络无关操作)

用例

操作链

断言

① 关于手机

open_settings(panel="about_phone") → get_screen

读到设备信息元素(手机名称/存储空间/运行内存/电池)

② 打开 App

open_app("设置")

screen_changed=true,首页出现搜索框等关键元素

③ 表单填写

定位输入框 → type_text → press_key("enter") → get_screen

搜索结果出现(如输入 wifi 出现 WLAN 相关项)

验收用例固化为 tests/test_integration.py,通过 MCP Client 边界端到端执行(ANDROID_TEST_DEVICE=<serial> 时启用)。

开发与测试

.venv/bin/pytest                                    # 单测(无设备,集成用例自动跳过)
.venv/bin/pytest tests/test_integration.py -v       # 验收用例(需设备)
ANDROID_TEST_DEVICE=192.168.1.15:39455 .venv/bin/pytest   # 全量含真机探针

路线图

  • 阶段 0 MVP:FastMCP + uiautomator2 + 语义动作 + 屏幕哈希 + 验证闭环 ✅

  • 阶段 1 通用性:OCR 融合感知 + 校验工具集(wait_for/verify_element/diff_state/smart_scroll)+ 多设备并行 ✅

  • 阶段 2 完整版:VLM 视觉融合(SoM) + 执行预算 + 增量 diff + AndroidWorld 子集评测 ✅ ← 当前

  • 后续:PyPI 发布(uv publish,token 就绪后)、端侧 companion App(独立 Android 工程)

Available Tools

18 tools
diff_stateA

Diff the current screen against the last cached snapshot.

Returns added/removed/text-changed element lists (stable-id diff) plus both screen hashes — the explicit counterpart of the automatic evidence returned by every action tool.

Inputs: device. Returns: object

get_device_infoA

Read device facts: model, manufacturer, Android version, resolution, density.

Inputs: device. Returns: object

get_screenA

Capture a compact text-mode snapshot of the current screen.

Returns elements as [id] class "label" kind@(x,y) lines plus the screen hash. Elements carry stable ids you can reference in tap().

Inputs: mode, device. Returns: object

get_screen_hashA

Return a cheap fingerprint of the current screen (hash of the snapshot).

Call this before and after an action to detect whether the screen changed, without refetching the full snapshot.

Inputs: mode, device. Returns: object

list_devicesA

List all adb-connected Android devices as {serial, connection_type}.

Returns an empty list (never an error) when no device is connected.

Inputs: no named parameters. Returns: result

locateA

Locate a described element via the VLM grounding layer (read-only).

The THIRD perception layer (tree -> OCR -> VLM): when the accessibility tree and OCR cannot resolve a target on a tree-failed screen (Flutter/ Unity/games), this asks a vision-language model for the element's bounding box on the current screenshot.

Inputs: target (required), device. Returns: object

open_appA

Launch an installed app by friendly name or package name.

Unknown/not-installed apps return a structured APP_NOT_INSTALLED error (with a hint to pass the exact package name).

Inputs: app_name (required), device, session_id. Returns: object

open_settingsA

Open a system settings panel via ACTION deep link.

Supported panels: home, about_phone, wifi, bluetooth, display, sound, apps, storage, security, location, battery, network, date, accessibility, language, developer.

Inputs: panel, device, session_id. Returns: object

press_keyA

Press a system key: back, home, recent, menu, enter, power, volume_up, volume_down, clear.

Inputs: key (required), device, session_id. Returns: object

reset_sessionA

Reset the session: clear per-device snapshot caches and the execution budget counters (spec execution-budget).

Call this after an EXECUTION_BUDGET_EXCEEDED error to continue, or whenever element ids from get_screen have gone stale (a fresh observer is created on the next screen access).

Returns the reset budget state {action_count, elapsed_seconds, max_actions, max_seconds, exceeded}.

Inputs: session_id. Returns: object

scroll_pageB

Scroll exactly one viewport page (native one-page scroll).

Inputs: direction (required), device, session_id. Returns: object

scroll_untilA

Scroll until a semantic target appears (or the scroll reaches its end).

Uses one-viewport-page scrolling and stops as soon as the container boundary is reached (no useless swipes at the end).

Inputs: target (required), device, direction, session_id, +1 more. Returns: object

smart_scrollA

Scroll and AGGREGATE every matching element across the screens passed.

Unlike scroll_until (first hit), smart_scroll visits each screen in the scroll budget and returns ALL matches with their positions. Reuses the one-viewport-page scrolling + end detection; on tree-failed screens the OCR layer is aggregated for free.

Inputs: target (required), device, direction, session_id, +1 more. Returns: object

swipeA

Swipe the screen by direction (up/down/left/right).

Returns verification evidence {executed, screen_changed, changed_elements[], screen_hash}.

Inputs: direction (required), device, distance, session_id. Returns: object

tapA

Tap a semantic target and verify the result.

Prefer semantic selectors — never compute coordinates yourself:

  • by=text: exact text label, e.g. {"by": "text", "value": "保存"}

  • by=desc: content-desc label

  • by=id: a stable element id from get_screen

  • by=bounds: last resort, "x,y" or "l,t,r,b" (raw coordinates)

Returns verification evidence {executed, screen_changed, changed_elements[], screen_hash}, or SELECTOR_AMBIGUOUS with candidates[] when several elements match (retry with by=id).

Inputs: target (required), device, session_id. Returns: object

type_textA

Type text into an input field.

With target, the field is located semantically first (must be an editable field). Without target, the currently focused field is used. The field is cleared before typing.

Inputs: text (required), device, target, session_id. Returns: object

verify_elementA

Assert whether a semantic target exists on the current screen.

Structured boolean result — never raises for not-found or a text mismatch, so the model can branch on it directly.

Inputs: target (required), device, text_predicate. Returns: object

wait_forA

Block until a semantic target appears or disappears on screen.

Polls fresh snapshots every interval seconds until the condition holds or timeout expires (handles loading screens and transient UI). Never raises on timeout — returns satisfied: false + final hash.

Inputs: target (required), state, device, timeout, +1 more. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.7/5.0

    Scored across 18 tools

    Disambiguation4/5

    Most tools target distinct actions or resources, but the four scroll variants (swipe, scroll_page, scroll_until, smart_scroll) and multiple verification helpers (wait_for, verify_element, diff_state) occupy adjacent territory. The descriptions do enough to separate them, so only one or two selections could initially be confused.

    Naming Consistency4/5

    Names are mostly snake_case imperative verb phrases like get_screen, open_app, and press_key, but a few like wait_for, scroll_until, and smart_scroll break the strict verb_noun pattern. The overall style is still consistent enough that an agent can predict tool names.

    Tool Count3/5

    At 18 tools, the surface is on the heavy side; several scroll and screen-verification helpers could potentially be consolidated. Each tool has a purpose, but the sheer number makes selection harder and feels borderline for an agent.

    Completeness3/5

    Core UI automation is covered well: screen capture, actions, waiting, verification, and navigation. However, there are notable gaps around app lifecycle management (no install/uninstall/stop) and no explicit screenshot capture tool, which can create dead ends for some phone-control workflows.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers