android-phone-mcp-server
This server provides semantic control and observation of Android devices via ADB, using three perception layers (accessibility tree, OCR, and VLM) for reliable UI interaction. All actions return structured results with verification evidence and machine-readable errors. Write operations are disabled by default (enable via environment variables); execution budgets prevent runaway agents. Multi-device support is built in.
Device Management: List connected devices and get device info (model, manufacturer, Android version, resolution, density).
Screen Observation: Capture compact or full text-mode snapshots with stable element IDs; get screen hashes for change detection; diff current vs cached screen state; locate elements via VLM when accessibility tree and OCR fail.
Semantic Interactions: Tap by text, content description, element ID, or coordinates; swipe in any direction with short/long distance; scroll exactly one viewport page; type text into focused or targeted fields; scroll until a target appears; aggregate matching elements across multiple screens (including OCR) with
smart_scroll; open apps by name or package; deep-link into system settings panels; simulate system key presses (back, home, volume, etc.).Verification & Waiting:
wait_forpolls until an element appears or disappears;verify_elementchecks existence and optionally a text predicate; all actions return verification evidence (changed elements, screen hash).Session Management: Reset session caches and execution budget counters.
Key Features: Read-only by default; execution budgets on action count/time; structured errors with hints; three-layer perception; full verification feedback for agent reliability.
Provides semantic control of Android devices via ADB, including tap, swipe, scroll, text input, opening apps and settings, and screen observation with verification.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@android-phone-mcp-server Open the Settings app and go to the About phone page"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
android-phone-mcp
代码 Agent 无关的 Android 控制 MCP Server:语义动作 + 验证闭环 + 融合感知。 Any MCP client (Claude Code / Cursor / Cline / 自研 Agent) can control an Android device through semantic tools — no coordinate guessing, every action returns verification evidence.
✅ Phase 0(语义动作 + 验证闭环)/ Phase 1(OCR 融合感知 + 校验工具集 + 多设备并行)/ Phase 2(VLM 视觉融合 + 执行预算 + 增量 diff + 评测)均已完成真机验收。 完整设计见
android-phone-mcp-server-设计文档.md与openspec/。
快速开始(开发环境)
# 1. 创建虚拟环境并安装(uv;国内网络请配置镜像)
uv venv .venv
export UV_DEFAULT_INDEX=https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple # 可选
uv pip install -e ".[dev]"
# 2. 连接 WiFi adb 设备
adb connect <phone-ip>:<port> # 例如 192.168.1.15:39455
adb devices # 确认 device 状态
# 3. 启用开发写权限(默认只读!)
cp .env.example .env
# 4. 启动 MCP server(stdio)
.venv/bin/android-phone-mcp --stdio
# 5. 常用检查
.venv/bin/android-phone-mcp --show-config # 查看生效配置
.venv/bin/android-phone-mcp --list-tools # 列出全部 18 个工具作为 MCP 客户端接入(Python 示例)
import asyncio
from fastmcp import Client
from android_phone_mcp.config import Config
from android_phone_mcp.server import create_server
async def main():
server = create_server(config=Config(allow_write=True)) # 开发期开写
async with Client(server) as client:
r = await client.call_tool("open_settings", {"panel": "about_phone"})
print(r.data) # {executed, screen_changed, changed_elements[], screen_hash}
asyncio.run(main())任意 MCP 客户端(Claude Code / Cursor / Cline / 自研 Agent)均可通过 MCP 协议接入;错误以结构化 JSON 返回(如 WRITE_DISABLED / SELECTOR_AMBIGUOUS + candidates[]),模型可直接读取提示继续操作。
Related MCP server: uiautomator2 MCP Server
安全模型
开关 | 环境变量 | 默认 |
写操作(tap/输入/安装等) |
| off(只读) |
任意 |
| off |
每动作超时 |
| 30s |
配置优先级:环境变量 > config.yaml > 内置默认。.env 仅用于本地开发一键开写。
工具清单(18 个)
类别 | 工具 |
设备 |
|
观察 |
|
动作 |
|
等待 |
|
会话/护栏 |
|
校验工具(阶段 1 新增):
wait_for(target, state=present|absent, timeout=10):阻塞等待元素出现/消失(加载动画)verify_element(target, text_predicate?):断言元素存在/缺失/文本匹配,结构化布尔diff_state():当前 vs 上次快照的增量 diff(added/removed/changed + 双 hash)smart_scroll(target):滚动聚合多屏全部命中(含 OCR 感知),一次调用返回
OCR 融合感知(阶段 1)
无障碍树对 Flutter/Unity/游戏等自渲染引擎失效时,get_screen 自动回退 OCR 文本层:识别文本以 ocr- 前缀伪元素合并进紧凑快照(共享 id 空间),tap/type_text 可直接定位;tap 前会重新 OCR 确认位置(OCR_STALE 兜底)。
# 安装 OCR 引擎(可选 extra,默认不装)
uv pip install -e ".[ocr]" # 或 pip install android-phone-mcp[ocr]
# 引擎:rapidocr_onnxruntime(py3.14 无 paddlepaddle wheel 时的替代,中文识别佳)配置 | 默认 | 说明 |
| true | OCR 兜底开关 |
| 3 | 树可交互元素低于该值触发 OCR |
VLM 视觉融合(阶段 2 · 第三感知层)
当无障碍树与 OCR 都无法定位目标时(抽象图形界面/游戏 UI),可选 VLM grounding(OpenAI 兼容视觉端点,如 Qwen2.5-VL via vLLM/Ollama)作为最终兜底:
locate(target):按文本/描述返回目标位置{found, position, box}自动兜底:语义工具(tap/scroll_until 等)目标未命中且 VLM 可用时,自动 VLM 定位,
vlm-伪元素进快照(共享 id 空间);tap 前重确认(VLM_STALE兜底)SoM 编号截图通道:vision 模型可直接看编号截图操作
# 安装 VLM 引擎(可选 extra,默认不装)
uv pip install -e ".[vlm]" # 或 pip install android-phone-mcp[vlm]配置 | 默认 | 说明 |
| true | VLM 兜底开关 |
| 空 | OpenAI 兼容端点(如 |
| 空 | 模型名(如 |
| 空 | 本地端点可留空 |
VLM 为服务端能力——调用模型不需要视觉;未配置端点时结构化降级(
VLM_UNAVAILABLE),不影响树/OCR 路径。
执行预算(阶段 2 · 防失控护栏)
会话级动作数/时长上限,防止 Agent 死循环/狂点:
配置 | 默认 | 说明 |
| 0(关) | 最大写动作数,超出拒绝 |
| 0(关) | 自首个动作起最大时长 |
超限后写工具返回 EXECUTION_BUDGET_EXCEEDED(含 limit 与恢复提示);调用 reset_session 清除快照缓存并重置预算:
tap(...) # ok
tap(...) # 第 N+1 次 -> {error: "EXECUTION_BUDGET_EXCEEDED", limit: "actions", hint: "..."}
reset_session() # {ok: true, budget: {action_count: 0, ...}}
tap(...) # ok(预算已重置)AndroidWorld 子集评测(阶段 2)
# 运行评测(需要 WiFi adb 设备)
ANDROID_TEST_DEVICE=<serial> .venv/bin/python -m eval.runner
# 输出 eval/report.json:每任务 {task, success, steps, error/reason}WiFi adb 注意事项与重连流程
WiFi adb 注意事项与重连流程
调试设备通过 WiFi adb 连接时,请勿在测试中切换 Wi-Fi 开关——关闭瞬间手机会断网,adb 连接随之断开
IP:端口会随重连变化;灭屏/省电可能导致连接离线
重连流程:
# 1. 确认离线
adb devices # 设备消失或显示 offline
# 2. 断开旧连接并重新 connect(手机端需保持"无线调试"开启)
adb disconnect
adb connect <phone-ip>:<port>
# 3. 验证
adb devices # 应显示 device(非 offline)
adb -s <serial> shell getprop ro.product.model # 输出模型号即正常若
connect后反复 offline:检查手机无线调试端口是否变化、与电脑是否同网段、防火墙是否放行 5555 段端口设备池在每次工具调用前做 health check,连接失效时返回结构化错误,不会卡死调用
验收用例(Phase 0 三用例,均为网络无关操作)
用例 | 操作链 | 断言 |
① 关于手机 |
| 读到设备信息元素(手机名称/存储空间/运行内存/电池) |
② 打开 App |
|
|
③ 表单填写 | 定位输入框 → | 搜索结果出现(如输入 wifi 出现 WLAN 相关项) |
验收用例固化为 |
开发与测试
.venv/bin/pytest # 单测(无设备,集成用例自动跳过)
.venv/bin/pytest tests/test_integration.py -v # 验收用例(需设备)
ANDROID_TEST_DEVICE=192.168.1.15:39455 .venv/bin/pytest # 全量含真机探针路线图
阶段 0 MVP:FastMCP + uiautomator2 + 语义动作 + 屏幕哈希 + 验证闭环 ✅
阶段 1 通用性:OCR 融合感知 + 校验工具集(wait_for/verify_element/diff_state/smart_scroll)+ 多设备并行 ✅
阶段 2 完整版:VLM 视觉融合(SoM) + 执行预算 + 增量 diff + AndroidWorld 子集评测 ✅ ← 当前
后续:PyPI 发布(uv publish,token 就绪后)、端侧 companion App(独立 Android 工程)
Available Tools
18 toolsdiff_stateA
Diff the current screen against the last cached snapshot.
Returns added/removed/text-changed element lists (stable-id diff) plus both screen hashes — the explicit counterpart of the automatic evidence returned by every action tool.
Inputs: device. Returns: object
get_device_infoA
Read device facts: model, manufacturer, Android version, resolution, density.
Inputs: device. Returns: object
get_screenA
Capture a compact text-mode snapshot of the current screen.
Returns elements as [id] class "label" kind@(x,y) lines plus the
screen hash. Elements carry stable ids you can reference in tap().
Inputs: mode, device. Returns: object
get_screen_hashA
Return a cheap fingerprint of the current screen (hash of the snapshot).
Call this before and after an action to detect whether the screen changed, without refetching the full snapshot.
Inputs: mode, device. Returns: object
list_devicesA
List all adb-connected Android devices as {serial, connection_type}.
Returns an empty list (never an error) when no device is connected.
Inputs: no named parameters. Returns: result
locateA
Locate a described element via the VLM grounding layer (read-only).
The THIRD perception layer (tree -> OCR -> VLM): when the accessibility tree and OCR cannot resolve a target on a tree-failed screen (Flutter/ Unity/games), this asks a vision-language model for the element's bounding box on the current screenshot.
Inputs: target (required), device. Returns: object
open_appA
Launch an installed app by friendly name or package name.
Unknown/not-installed apps return a structured APP_NOT_INSTALLED error (with a hint to pass the exact package name).
Inputs: app_name (required), device, session_id. Returns: object
open_settingsA
Open a system settings panel via ACTION deep link.
Supported panels: home, about_phone, wifi, bluetooth, display, sound, apps, storage, security, location, battery, network, date, accessibility, language, developer.
Inputs: panel, device, session_id. Returns: object
press_keyA
Press a system key: back, home, recent, menu, enter, power, volume_up, volume_down, clear.
Inputs: key (required), device, session_id. Returns: object
reset_sessionA
Reset the session: clear per-device snapshot caches and the execution budget counters (spec execution-budget).
Call this after an EXECUTION_BUDGET_EXCEEDED error to continue, or whenever element ids from get_screen have gone stale (a fresh observer is created on the next screen access).
Returns the reset budget state {action_count, elapsed_seconds, max_actions, max_seconds, exceeded}.
Inputs: session_id. Returns: object
scroll_pageB
Scroll exactly one viewport page (native one-page scroll).
Inputs: direction (required), device, session_id. Returns: object
scroll_untilA
Scroll until a semantic target appears (or the scroll reaches its end).
Uses one-viewport-page scrolling and stops as soon as the container boundary is reached (no useless swipes at the end).
Inputs: target (required), device, direction, session_id, +1 more. Returns: object
smart_scrollA
Scroll and AGGREGATE every matching element across the screens passed.
Unlike scroll_until (first hit), smart_scroll visits each screen in the scroll budget and returns ALL matches with their positions. Reuses the one-viewport-page scrolling + end detection; on tree-failed screens the OCR layer is aggregated for free.
Inputs: target (required), device, direction, session_id, +1 more. Returns: object
swipeA
Swipe the screen by direction (up/down/left/right).
Returns verification evidence {executed, screen_changed, changed_elements[], screen_hash}.
Inputs: direction (required), device, distance, session_id. Returns: object
tapA
Tap a semantic target and verify the result.
Prefer semantic selectors — never compute coordinates yourself:
by=text: exact text label, e.g. {"by": "text", "value": "保存"}
by=desc: content-desc label
by=id: a stable element id from get_screen
by=bounds: last resort, "x,y" or "l,t,r,b" (raw coordinates)
Returns verification evidence {executed, screen_changed, changed_elements[], screen_hash}, or SELECTOR_AMBIGUOUS with candidates[] when several elements match (retry with by=id).
Inputs: target (required), device, session_id. Returns: object
type_textA
Type text into an input field.
With target, the field is located semantically first (must be an editable field). Without target, the currently focused field is used. The field is cleared before typing.
Inputs: text (required), device, target, session_id. Returns: object
verify_elementA
Assert whether a semantic target exists on the current screen.
Structured boolean result — never raises for not-found or a text mismatch, so the model can branch on it directly.
Inputs: target (required), device, text_predicate. Returns: object
wait_forA
Block until a semantic target appears or disappears on screen.
Polls fresh snapshots every interval seconds until the condition
holds or timeout expires (handles loading screens and transient UI).
Never raises on timeout — returns satisfied: false + final hash.
Inputs: target (required), state, device, timeout, +1 more. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 18 tools
Most tools target distinct actions or resources, but the four scroll variants (swipe, scroll_page, scroll_until, smart_scroll) and multiple verification helpers (wait_for, verify_element, diff_state) occupy adjacent territory. The descriptions do enough to separate them, so only one or two selections could initially be confused.
Names are mostly snake_case imperative verb phrases like get_screen, open_app, and press_key, but a few like wait_for, scroll_until, and smart_scroll break the strict verb_noun pattern. The overall style is still consistent enough that an agent can predict tool names.
At 18 tools, the surface is on the heavy side; several scroll and screen-verification helpers could potentially be consolidated. Each tool has a purpose, but the sheer number makes selection harder and feels borderline for an agent.
Core UI automation is covered well: screen capture, actions, waiting, verification, and navigation. However, there are notable gaps around app lifecycle management (no install/uninstall/stop) and no explicit screenshot capture tool, which can create dead ends for some phone-control workflows.
Maintenance
Related MCP Connectors
Melaya is a remote MCP server. It gives an assistant hands on your own Android phone and browser: it reads the screen through the accessibility tree, then taps, types and navigates inside the apps and sites you allow-list, with no per-app API. It also builds, schedules and runs agent pipelines across 6k+ connected tools. OAuth 2.1, nothing to install.
MCP server for your apps' tools and custom tools, plus hosted AI agents and approval-gated workflows
MCP server for AI agents: posts, messages and comments on WhatsApp, Instagram, TikTok and Facebook.
ORDan's MCP server: calls, SMS and phone numbers for AI agents, under their owner's spend controls
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA MCP server that enables LLMs to control Android devices via ADB, supporting input, UI hierarchy, device management, and shell commands.16MIT
- AlicenseNot gradedqualityBmaintenanceAn MCP server that provides tools for controlling Android devices using uiautomator2, enabling AI to automate tasks like tapping, swiping, and managing apps.186 PyPI57Apache 2.0
- AlicenseNot gradedqualityDmaintenanceMCP server that enables LLMs to control Android devices via ADB, providing tools for screen interaction and UI inspection.1MIT
- AlicenseNot gradedqualityAmaintenanceAn MCP server that enables AI agents to control Android and iOS devices via natural language, using platform tools like adb and simctl.3,290 npm57Apache 2.0