Skip to main content
Glama
multi_uiautomator2

multi_uiautomator2

by Starlordzz

Multi-UIAutomator2 (Automation Service & SDK)

按需启动、无常驻后台、多语言共享(Python / Dart / MCP)、带 AI 视觉与语义理解的 Android UI 自动化服务


⚠️ 重复造轮子声明

本项目是一个典型的重复造轮子项目 —— 市面上已有许多更成熟、更完善的同类方案,如果没有特殊的自用需求,请优先使用它们:

已有方案

说明

openatx/uiautomator2

本项目底层依赖的 Python 库,单项目使用直接用它即可

mobile-next/mobile-mcp

让 AI 客户端(Claude / Cursor 等)直接操控 Android / iOS 设备的 MCP Server

droidrun/droidrun

LLM 驱动的 Android 自动化 Agent 框架

Appium

老牌跨平台移动端自动化框架

Maestro

简单易用的移动端 UI 测试框架

本项目继续存在的理由只有两个:

  • 学习练手:把 uiautomator2、MCP 协议、多语言 SDK 串起来完整跑通一遍;

  • 自用场景:多个本地项目共享同一个「按需启动、空闲自退出」的自动化服务实例,并顺带暴露为 MCP 工具。

不追求生产可用,不承诺持续维护。


Related MCP server: Android MCP Server

🤖 MCP Server(AI 客户端直连安卓设备)

本项目内置 MCP Server(mcp_server/),把全部自动化能力(观察 / 点按 / 输入 / 截图 / SoM 视觉标注 / AI 自愈)暴露为 24 个 MCP 工具,可在 ZCode、Claude Desktop、Cursor 等 AI 客户端中直接操控安卓设备:

{
  "mcpServers": {
    "android-device": {
      "command": "uv",
      "args": ["--directory", "/path/to/multi_uiautomator2", "run", "mcp-server-android"]
    }
  }
}

安装与配置详见 mcp_server/README.md。


🎯 核心设计理念

  1. 按需启动,用完即走(No Always-Running Daemon):

    • 本地平时没有任何常驻进程占用 CPU / 内存 / 端口。

    • 任何 Python、Dart 或其他子项目调用自动化接口时,SDK 自动在后台静默启动 automation-server。

    • Idle Watchdog(空闲看门狗):默认 60 秒无新请求时,服务自动断开设备连接并退出进程。

  2. 多项目统一,彻底杜绝冲突:

    • 解决多项目各自依赖 uiautomator2 导致的 ADB 抢占、端口冲突、Session 冲突。

    • Dart / Flutter / Python / JS 等所有项目只依赖极轻量的 Client SDK。

  3. AI 视觉与语义理解(Self-Understanding & AI Debug):

    • 自动将庞大的 Android XML 结构树提炼为编号交互元素列表(大幅降低 90%+ Token 消耗)。

    • 支持 Set-of-Marks (SoM) 视觉标记与多模态大模型对接。

    • 提供自然语言目标驱动 (ai_act)、界面语义分析 (ai_understand) 和异常弹窗/崩溃自动诊断修复 (ai_debug)。


🏗️ 架构全景

 ┌──────────────────────┐      ┌──────────────────────┐
 │    Python 项目       │      │     Dart / Flutter   │
 │ (automation_client)  │      │ (automation_client)  │
 └──────────┬───────────┘      └──────────┬───────────┘
            │                             │
            │ HTTP (离线时自动拉起 Server)  │
            └──────────────┬──────────────┘
                           ↓
        ┌──────────────────────────────────────┐
        │       Automation Server (8765)       │
        │  - 60s 空闲自动退出 (Idle Watchdog)   │
        │  - UIAutomator2 / ADB 设备池调度     │
        │  - UI 树精简与 Set-of-Marks 视觉标记  │
        │  - AI 目标规划与自动调试诊断          │
        └──────────────────┬───────────────────┘
                           │
             ┌─────────────┴─────────────┐
             ↓                           ↓
            ADB                     UIAutomator2
             │                           │
             └─────────────┬─────────────┘
                           ↓
                    Android 设备 / 模拟器

📦 项目结构

multi_uiautomator2/
├── server/                   # 自动化服务端核心
│   ├── app.py                # FastAPI 入口 & CLI
│   ├── config.py             # 配置管理 (端口、空闲超时、LLM 密钥)
│   ├── lifecycle.py          # Idle Watchdog 空闲自动退出看门狗
│   ├── core/
│   │   ├── device_manager.py # ADB 与 uiautomator2 连接与会话管理
│   │   ├── actions.py        # 统一动作引擎 (点击/输入/滑动/按键/App生命周期)
│   │   ├── hierarchy.py      # UI 树解析与 AI 编号精简器
│   │   └── screenshot.py     # 截图与 Set-of-Marks (SoM) 视觉标记
│   ├── ai/
│   │   ├── agent.py          # 界面理解、多步目标执行与自愈调试
│   │   └── llm_provider.py   # OpenAI / Gemini / 本地大模型统一接口
│   └── routes/               # API 路由 (/device, /app, /ui, /ai)
├── mcp_server/               # MCP Server (AI 客户端 stdio 接入, 24 个工具)
│   ├── server.py
│   └── README.md
├── sdk/
│   ├── python/               # Python SDK
│   │   ├── automation_client/
│   │   └── examples/
│   └── dart/                 # Dart SDK
│       ├── lib/
│       └── example/
├── tests/                    # 单元与集成测试
└── requirements.txt          # Python 依赖

🚀 快速开始

1. 安装服务端依赖

pip install -r requirements.txt

2. Python 项目中使用

在 Python 项目中引入客户端即可,无需手动启动服务端:

from automation_client import AutomationClient

# 创建 Client(会自动检测并在后台拉起 server,60s 无请求后自动关闭)
client = AutomationClient(auto_start_server=True, idle_timeout=60)

# 连接手机
client.connect()

# 基础操作
client.click(text="登录")
client.input(text="my_username", resource_id="com.example:id/username")
client.press("home")

# 截图保存
client.screenshot(save_path="screen.png")

# 获取精简 UI 树
dump = client.dump(simplified=True)
print(f"当前屏幕有 {dump['element_count']} 个可操作控件")

# AI 自然语言驱动(需配置 OPENAI_API_KEY)
res = client.ai_act(goal="点击设置图标进入关于手机页面")
print("AI 执行结果:", res)

3. Dart / Flutter 项目中使用

在 Dart / Flutter 项目中:

import 'package:automation_client/automation_client.dart';

void main() async {
  // 创建 Client(Dart 端纯 HTTP,自动检测并按需拉起本地服务)
  final client = AutomationClient(autoStartServer: true, idleTimeout: 60);

  // 连接设备
  await client.connect();

  // 点击与输入
  await client.click(text: "登录");
  await client.input(text: "my_username", resourceId: "username");

  // 截屏保存
  await client.screenshot(savePath: "screen.png");

  // 按键
  await client.press(key: "home");

  // AI 语义分析当前界面
  final analysis = await client.aiUnderstand();
  print("页面类型: ${analysis['analysis']['screen_type']}");

  client.close();
}

🧠 AI Agent 与自我理解能力

功能

API

说明

语义理解

POST /ai/understand

识别当前页面类型 (登录/表单/列表/弹窗),提炼核心控件

目标驱动

POST /ai/act

接受自然语言目标(如 "打开WiFi设置并开启开关"),自动进行「观察 -> 思考 -> 点击/输入 -> 验证」多步闭环

异常自愈

POST /ai/debug

检测界面是否发生 Crash、ANR、权限拦截或异常遮罩,并可自动点击关闭或回退


🛠️ REST API 接口清单

分组

方法

路径

描述

System

GET

/health

健康检查 (SDK 检测存活)

GET

/status

服务状态与空闲倒计时

POST

/config/idle-timeout

动态修改空闲超时时间 (秒)

POST

/shutdown

手动关闭服务

Device

GET

/device/list

列出所有 ADB 设备

POST

/device/connect

连接指定/默认设备

GET

/device/info

获取分辨率/前台App等信息

POST

/device/disconnect

断开并释放设备会话

UI

POST

/ui/click

点击坐标或选择器

POST

/ui/input

输入文本

POST

/ui/clear

清除输入框文本

POST

/ui/swipe

屏幕滑动

POST

/ui/scroll

页面滚动 (up/down/left/right)

POST

/ui/press

按键 (back/home/enter 等)

POST

/ui/screenshot

截屏 (可附带 SoM 视觉标注)

POST

/ui/dump

获取精简 UI 树与交互元素

POST

/ui/find

检查元素是否存在

POST

/ui/wait

等待元素出现

App

POST

/app/start

启动指定 App

POST

/app/stop

强制停止 App

GET

/app/current

获取当前前台 App

AI

POST

/ai/understand

屏幕语义解析

POST

/ai/act

自然语言多步目标执行

POST

/ai/debug

异常弹窗/崩溃诊断与自动修复

Available Tools

24 tools
ai_debug_screenA

Diagnose the current screen for crashes, ANRs, permission blockers or unexpected overlays and report findings; with auto_fix=true it also attempts recovery. Requires OPENAI_API_KEY (or LLM_* env).

Inputs: auto_fix. Returns: result

ai_execute_goalA

Delegate a natural-language goal to the server-side AI agent, which runs its own observe-think-act loop on the device and returns a step log. Needs OPENAI_API_KEY (or LLM_API_KEY / OPENAI_BASE_URL / LLM_MODEL). Prefer the primitive tools (get_screen_elements + tap + input_text) unless the user explicitly asks to delegate.

Inputs: goal (required), max_steps. Returns: result

ai_understand_screenA

Ask the server-side LLM to semantically analyze the current screen: page type (login/form/list/dialog), purpose and key elements. Requires OPENAI_API_KEY (or LLM_API_KEY / OPENAI_BASE_URL / LLM_MODEL) to be configured; otherwise prefer get_screen_elements + screenshot and analyze yourself.

Inputs: no named parameters. Returns: result

clear_app_dataA

DESTRUCTIVE: clear all user data of an app (logs the user out, wipes settings). Requires confirm=true.

Inputs: package_name (required), confirm. Returns: result

clear_textC

Clear the text of an input field found by selector, or of the currently focused field when no selector is given.

Inputs: index, resource_id, target_text, text_contains. Returns: result

connect_deviceA

Connect to an Android device and remember it as the default for all subsequent tool calls in this session. Returns screen size and current app.

Inputs: serial. Returns: result

disconnect_deviceA

Disconnect a device and release its uiautomator2 session. Omit serial to disconnect the current default device.

Inputs: serial. Returns: result

double_tapB

Double-tap at (x, y) or on an element found by selector (text / resource_id / index).

Inputs: x, y, text, index, +1 more. Returns: result

find_elementA

Check whether an element matching the selector currently exists on screen. Provide at least one selector field.

Inputs: text, index, description, resource_id, +1 more. Returns: result

get_device_infoA

Get info about the current device: serial, screen resolution, Android SDK level, screen-on state and the foreground app.

Inputs: no named parameters. Returns: result

get_screen_elementsA

Dump the current screen as a compact numbered list of interactive elements: [index] text="..." id="..." [flags] center=(x,y). This is the cheapest way to observe the screen; use the center coordinates with tap(x, y) or the text/resource_id with selector-based actions.

Inputs: raw_json. Returns: result

get_server_statusA

Get automation-server status: uptime, idle-watchdog countdown and currently connected devices. The underlying server auto-starts on demand and exits after the idle timeout.

Inputs: no named parameters. Returns: result

input_textB

Type text into an input field found by selector, or into whatever field currently has focus when no selector is given.

Inputs: content (required), clear, index, resource_id, +2 more. Returns: result

list_devicesA

List all Android devices attached via ADB, with their online state and whether a uiautomator2 session is already established.

Inputs: no named parameters. Returns: result

long_pressA

Long-press at (x, y) or on an element found by selector (context menus, selection mode, drag handles).

Inputs: x, y, text, index, +3 more. Returns: result

press_keyA

Press a device key: home, back, enter, delete, recent (app switcher), menu, search, volume_up/down, power, camera, or D-pad keys.

Inputs: key (required). Returns: result

screenshotA

Capture the current screen as an image so you can visually verify state. Returns the image plus a text summary. Combine with_som=true with get_screen_elements to map numbered boxes to element indices.

Inputs: max_dim, with_som, save_path.

scrollA

Scroll the current scrollable view in a direction without needing coordinates.

Inputs: direction. Returns: result

shutdown_automation_serverA

Manually stop the background automation server to free resources. It is re-spawned automatically on the next tool call, so this is always safe.

Inputs: no named parameters. Returns: result

start_appB

Launch an app on the current device by package name.

Inputs: package_name (required), activity, stop_first. Returns: result

stop_appA

Force-stop an app by package name.

Inputs: package_name (required). Returns: result

swipeC

Swipe/drag from (sx, sy) to (ex, ey). Get the screen size from get_device_info to compute coordinates.

Inputs: ex (required), ey (required), sx (required), sy (required), +1 more. Returns: result

tapB

Tap the screen either at (x, y) coordinates or on an element found by selector. Provide coordinates OR at least one selector field.

Inputs: x, y, text, index, +5 more. Returns: result

wait_for_elementB

Block until an element matching the selector appears (e.g. after tapping a button, wait for the next page). Provide at least one selector field.

Inputs: text, index, timeout, resource_id, +1 more. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 24 tools

    Disambiguation4/5

    Tools are largely distinct, but there is some overlap between screenshot/get_screen_elements (both observe the screen) and among the AI-powered tools (ai_debug_screen vs ai_understand_screen vs ai_execute_goal). Descriptions clarify differences, but an agent might occasionally misselect between observation and AI analysis tools.

    Naming Consistency4/5

    Most tool names follow a verb_noun pattern (e.g., list_devices, start_app, input_text). The main deviation is 'screenshot', which is a noun without an explicit verb, and 'press_key' could be seen as slightly inconsistent but still clear. Overall, the naming is predictable and readable.

    Tool Count3/5

    With 24 tools, the server is on the heavier side (16-25 is considered heavy). However, the count is justified by the breadth of features: device management, app lifecycle, UI interactions, observation, and AI delegation. Still, some tools (like the three AI ones) might be consolidated, making the set feel slightly bloated.

    Completeness4/5

    The toolset covers the core lifecycle: device connection, app control, UI interaction (tap, swipe, scroll, input), observation (screenshot, get_screen_elements), and waiting/finding elements. It lacks explicit handling of system dialogs or advanced gestures, but the AI tools can cover some gaps. Overall, no critical dead ends for typical automation workflows.

    Maintenance

    ActivityStale
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      An MCP server that provides tools for controlling Android devices using uiautomator2, enabling AI to automate tasks like tapping, swiping, and managing apps.
      186 PyPI
      57
      Apache 2.0
    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables agents to interact with Android via text-based UI trees instead of screenshots, supporting taps, swipes, input, macro recording, and device control through an MCP server and CLI.
      1
      -
    • A
      license
      Not graded
      quality
      A
      maintenance
      Gives MCP-compatible AI assistants vision and control over Android devices — capturing screenshots, tapping, swiping, typing, launching apps, inspecting UI elements, streaming audio/video, transferring files, and running shell commands over ADB, with a fast binary control protocol for near-instant input and screenshots.
      392 npm
      MIT