Skip to main content
Glama

mobile-agent-harness

Android device automation for AI agents. MCP server, CLI, and a hot-reloadable plugin runtime on top of a self-healing uiautomator2 bridge. No device-side app required.

中文文档:README.zh-CN.md

This is the base runtime only. Plugins live in the community catalog: awesome-mobile-agent-harness-plugin. App business knowledge for AI lives in app-lore, which this repo's knowledge/ loader reads directly.

Install

git clone https://github.com/Aelindra/mobile-agent-harness
cd mobile-agent-harness
pip install -e .          # add ".[ocr]" for vision.ocr / ui2.*

Requires Python 3.9+ and adb on PATH with USB debugging enabled.

Related MCP server: android-mcp-tools

Quick start

  1. Point the bridge at your device:

export AGENT_SERIAL_DEFAULT=192.168.1.23:5555   # or "emulator-5555"
  1. List the tool surface (works without a device) and run the offline tests:

python -m core.cli tools
python tests/test_runtime.py
  1. Call a tool:

python -m core.cli call ui.snapshot
python -m core.cli call ui.click --args '{"selector":"text@^Network & internet$","pkg":"com.android.settings"}'
  1. Attach to any MCP client:

{
  "mcpServers": {
    "phone": {
      "command": "python",
      "args": ["-m", "core.cli", "mcp"],
      "cwd": "/path/to/mobile-agent-harness",
      "env": { "AGENT_SERIAL_DEFAULT": "192.168.1.23:5555" }
    }
  }
}

All tools return a {"ok": bool, "error"?, ...} envelope. ok=false is a business result (e.g. error_type=selector_not_found) — adapt instead of retrying.

Tools

Group

Tools

Observation

ui.snapshot ui.dump ui.find ui2.state ui2.screen_evidence ui2.timeline ui2.find_templates vision.screenshot vision.ocr vision.diff

Action

ui.click ui.click_handle ui.text_handle ui.hold_read ui.set_text ui.scroll ui.back ui.home input.tap input.swipe input.key

App

app.launch app.current app.probe app.list app.stop app.wait_idle

Shell & data

shell.run state.prefs state.db file.push file.pull net.http

Events

events.tail events.wait events.capture events.record_start events.record_stop

Knowledge & tasks

knowledge.list knowledge.guide ui2.check_states ui2.orient task.begin task.note task.end template.add sys.capabilities

ui.snapshot returns a compact accessibility-tree listing with element handles (e0, e1, ...); ui.click_handle acts on a handle with drift detection. ui2.state fuses the accessibility tree with OCR for canvas-drawn UIs. ui2.screen_evidence collects measurable screen features (overlay level, motion, color, layout density) without interpreting them. app.probe returns deterministic side signals (package, version, orientation); ui2.timeline captures N frames of lightweight evidence in time order; ui2.orient probes context, routes to matching knowledge packs, and evaluates their state discriminators — returning matched states or ranked hypotheses with the missing evidence listed.

Selectors

Selectors resolve against the live accessibility tree. Coordinates are not used.

# ui.click selector examples
"id@d98 && pkg@com.example.app"       # resource-id, short form completed with pkg@
"text@^Settings$ && pkg@com.example.app"
"class@android.widget.EditText && [clickable=true]"
"desc@^Search$"                        # content-desc regex

Harnesses and plugins

A harness file declares per-app flows as selector steps with pre/postconditions; distill drafts one by exploring an app, lint checks selector quality, and failed postconditions mark tools stale for re-distillation. See harnesses/com.android.settings/harness.json5 for the built-in example.

Plugins drop into plugins/ and register tools or capability providers at load time; registrations are reversible and hot-reloaded on file change. See the Plugin development guide in README.zh-CN.md and knowledge/README.md for the knowledge-pack format.

For business context on apps where model priors are unreliable, point agents at app-lore — a community catalog of structural app surveys in the open Agent Skills format, split per feature module for complex apps. Mobile scenarios in particular are private-domain and underrepresented in training data, so agents benefit from loading the relevant survey before operating an unfamiliar app. The local knowledge/ directory remains this repo's runtime layer (packs, icon templates, state discriminators).

Configuration

Variable

Description

Example

AGENT_SERIAL_DEFAULT

Device serial used when --serial/ANDROID_SERIAL absent

192.168.1.23:5555

AGENT_SERIAL_USB

USB serial fallback

0123456789abcdef

MAH_FRAME_STREAM

1 enables the H.264 frame stream for observation tools

1

MAH_VISION_MODE

Vision enhancement: off / client / endpoint

client

MAH_VLM_URL / MAH_VLM_MODEL

External vision endpoint (OpenAI-compatible)

http://127.0.0.1:11434/v1/chat/completions

How it compares

  • vs mobile-mcp / agent-device: they are generic device toolsets; this project adds the layer above transport — per-app knowledge as files (harnesses), a hot-reloadable plugin runtime with capability seams, and an explicit adb/root privilege model.

  • vs droidrun: droidrun requires a device-side accessibility app; this project is zero-install over adb.

  • Root support is declarative: if the device already has su, the shell seam gains a root provider gated by a command allowlist. This project does not provide root.

Known limitations

Messaging, banking, and e-commerce apps run active anti-automation risk control; the adb tier is unreliable on them. Target use is development, testing, and your own device workflows.

Contributing

Plugins are contributed to awesome-mobile-agent-harness-plugin; app business knowledge to app-lore — see their contributing guides.

License

MIT.

Available Tools

60 tools
app.currentB

当前前台应用

app.launchB

启动应用并等待前台空闲

Inputs: pkg (required).

app.listB

列出已安装应用包名(third_party=true 仅用户应用)

Inputs: filter, third_party.

app.stopD

停止应用

Inputs: pkg (required).

app.wait_idleC

等待应用前台且稳定

Inputs: pkg (required), timeout.

com.android.settings__launchA

[harness:com.android.settings] 打开设置并等待其稳定到前台

Inputs: no named parameters.

device.wake_unlockA

亮屏 + 按需解除非安全锁屏 + 清理残留弹窗(agent 自身 setup,非人工干预)

events.captureC

执行动作同时捕获瞬时 Toast/弹层(Monitor 接线;action=click 可带 selector;expect 为正则过滤)

Inputs: pkg, action, expect, timeout, +1 more.

events.record_startB

后台连拍+状态流开始(record_stop 收结果)

Inputs: interval.

events.record_stopA

停止后台记录,返回 toasts/帧数/目录

events.snapshotB

单次前台状态快照(含游离窗口节点=toast/悬浮窗候选)

events.tailB

事件面:最近的设备事件(page_change/app_start/app_crash/visual_change)——'刚才发生了什么'的场景感知,推送非轮询

Inputs: n, type.

events.waitA

等一个匹配事件(推送,非 sleep 轮询):type=page_change/app_start/app_crash/visual_change;可限定 pkg。事件到后仍应用帧/快照验证防误报

Inputs: type (required), pkg, timeout.

file.pullA

手机→PC 传文件:base64 读出解码(绕开 adb pull 的不可靠/权限问题),MD5 校验

Inputs: local (required), remote (required), root, max_mb.

file.pushB

PC→手机传文件:base64 分块管道(WiFi adb push 二进制不可靠的替代),MD5 校验

Inputs: local (required), remote (required), root, chunk.

harness.check_versionsA

读取设备上各 harness 目标 app 的 versionName,与 versionRange(>=/<=/* /通配) 比对;不匹配 → stale 需重蒸馏

harness.listB

列出已加载的 harness 与工具

harness.runC

运行某 harness 工具(steps 由语义选择器组成)

Inputs: pkg (required), tool (required), params, dry_run, +1 more.

input.clipboardB

剪贴板读写(set=写入设备剪贴板;get 读回,需 u2 支持)

Inputs: action (required), text.

input.keyC

按键注入(back/home/menu/volume_up/volume_down/power/enter... 或键码整数)

Inputs: key (required).

input.swipeC

裸坐标滑动(通用层原语)

Inputs: x1 (required), x2 (required), y1 (required), y2 (required), +1 more.

input.tapB

裸坐标点击(通用层原语:不解析选择器,直接注入;长按调大 duration_ms)

Inputs: x (required), y (required), duration_ms.

knowledge.guideA

读取知识包的 guide(上下文面:业务流程/使用时机/注意事项)。用某 app 前先看对应包的 guide

Inputs: pack (required).

knowledge.listB

列出已加载知识包(guide=上下文面,templates/states=运行时面)

net.httpC

发 HTTP 请求(proxy 参数可接 mitmproxy 等审计代理;verify 默认关闭以适配自签代理证书)

Inputs: url (required), body, proxy, method, +2 more.

shell.runA

执行 shell(root=true 走 su,设备需已 root);风险命令需 allow_risk。script=多行脚本模式:base64 推到手机 /data/local/tmp 执行(规避嵌套引号/管道符转义地狱),与 cmd 二选一

Inputs: cmd, root, script, allow_risk.

state.dbC

root 拉取应用 sqlite 库到 PC 临时目录并执行只读 SQL

Inputs: db (required), pkg (required), sql.

state.prefsC

root 读应用 shared_prefs XML 并解析

Inputs: pkg (required), file (required).

sys.capabilitiesB

能力发现:各缝当前可用 provider 及 privilege/stealth 元数据;插件热载入口

Inputs: reload_plugins.

task.beginA

开始一个业务任务:声明目标(如'领所有商品的券')。返回 task_id,后续 task.note/task.end 回传它

Inputs: goal (required).

task.endB

结束任务:实际结果与目标对照(成功/失败/原因)

Inputs: ok (required), outcome, task_id.

task.noteB

记录当前想法/判定/预期(业务化历史的核心):比如'我判断这个按钮是搜索入口,点击预期出现搜索框'。可能错,先声明后验证

Inputs: text (required), about, expect, task_id.

template.addA

声明图标模板:从截图(缺省最近一次 vision.screenshot)按 box 裁剪图标,写入 templates// 并更新 manifest。这是'图标→语义'声明层的一次性成本,之后 ui2.find_templates 永久可匹配(无需 hover)

Inputs: box (required), game (required), name (required), desc, +3 more.

ui2.check_statesB

按知识包的状态判别式求值当前屏幕:先采证据(同 ui2.screen_evidence),再对声明状态逐个判别式求值——全满足=ok,缺证据单列(不出具结论)。这是'证据与判读分离'的声明面

Inputs: game, pack.

ui2.diffC

界面变更检测:对比两次融合状态的文本元素增删

Inputs: gap.

ui2.find_templatesA

图标模板匹配:在当前屏幕上找声明图集里的图标(无需 hover,图标在屏即命中),返回位置+置信度+manifest 里声明的语义。templates 缺省匹配图集全部;图集用 template.add 从截图裁剪声明

Inputs: game, max_per, templates, threshold.

ui2.scene_matchA

场景/花纹零样本分类(小模型层):给候选标签短语(如 '设置页面'/'聊天列表页'),CLIP 类模型计算图文相似度返回各候选得分——加新场景=加一个标签,无需训练。定位:色彩/布局等像素统计概括不了的花纹与场景语义。需可选依赖:pip install 'mobile-agent-harness[scene]';模型经 MAH_SCENE_MODEL 配置(中文标签用 OFA-Sys/chinese-clip-vit-base-p16,英文标签默认 openai/clip-vit-base-patch32)

Inputs: candidates (required), box, top, from.

ui2.screen_evidenceB

屏幕证据采集(只出证据不下结论):overlay=灰屏/暗化覆盖(HSV)、motion=两帧运动量、color=色相分布、filter=vignette/flash、layout=UI 热区网格、hud=动作栏图标在位性(图集 kind=hud 模板)。防幻觉纪律:缺证据时列备选假设,不得凭单一特征断言状态

Inputs: game.

ui2.stateB

结构化 UI 状态 v2:a11y×OCR 融合——a11y 提供零误报可交互性,OCR 补 canvas/自绘文本(src=ocr 的元素为启发式推断)。元素带句柄,ui.click_handle 直接操作。canvas/自绘界面必用

Inputs: max_elements.

ui2.wait_changeB

等画面变化(点了按钮后确认生效),变化时返回新界面的可交互元素

Inputs: timeout, baseline_text.

ui.backC

系统返回键

ui.clickC

节点级点击(选择器解析,非裸坐标)

Inputs: selector (required), nth, pkg, timeout, +1 more.

ui.click_handleA

按快照句柄点击/长按(a11y 元素先重定位防漂移,纯视觉元素用快照中心)

Inputs: handle (required), long_click.

ui.dumpC

全量节点树 JSON(实时桥数据):class/id/text/desc/bounds/checked/clickable 等语义

Inputs: save_to, max_nodes.

ui.findC

语义查找节点(GKD 风格选择器),返回节点句柄摘要列表

Inputs: selector (required), pkg, timeout.

ui.get_textC

读取选择器命中节点的文本(只读不点击)

Inputs: selector (required), pkg, limit.

ui.hold_readA

长按读悬浮提示(hover/tooltip):后台线程长按目标,按住期间读屏对比,返回新出现的提示文本。适合悬浮说明、图标长按菜单等按住才显示的提示。ocr=false 只比 a11y 树(快,~1s,普通 app 用);ocr=true 加 OCR 对比(~6s,canvas/自绘界面用)

Inputs: x, y, ocr, pkg, +3 more.

ui.homeB

回到桌面

ui.long_clickC

节点级长按

Inputs: selector (required), pkg.

ui.scrollC

滚动(down=查看下方内容,手指上滑)

Inputs: pkg, container, direction.

ui.set_textB

精确输入到指定组件(优先 ACTION_SET_TEXT;无 selector 时输入到聚焦框)

Inputs: text (required), pkg, clear, selector.

ui.snapshotA

读屏默认首选:a11y 语义裁剪快照,元素编号 e0/e1/...(token 约为 ui.dump 的 1/10)。k 类型 btn/input/toggle/scroll/icon/text;i=可交互 e=可输入 s=可滚动 v=开关态。用 ui.click_handle/ui.text_handle 按句柄操作

Inputs: pkg, max_elements, include_bounds.

ui.text_handleB

按句柄向输入元素写文本(a11y 元素走 set_text 原生通道链;纯视觉元素先点中心聚焦)

Inputs: text (required), handle (required), clear.

ui.wait_forA

轮询等待选择器出现/消失(等弹窗、加载完成、页面跳转),超时返回 ok=false

Inputs: selector (required), pkg, gone, timeout.

vision.askA

通用识图增强入口(增强层,非主链路)。MAH_VISION_MODE 决定策略:client=截图以 MCP image 内容块返回,由具备视觉能力的客户端模型自己判读(零额外部署);endpoint=转发问题到外部部署的视觉端点(MAH_VLM_URL/MAH_VLM_MODEL,同 vision.vlm);off=关闭(默认,仅用 ui.snapshot/ui2.*/图集等确定性证据层)

Inputs: box, from, prompt.

vision.diffB

界面变化检测:当前截图 vs 基线的像素差异(变化占比+差异框);ocr=true 附文本增删

Inputs: ocr, current, baseline, threshold.

vision.ocrB

OCR(懒加载 rapidocr_onnxruntime;未安装则明确报错);path=本地图片离线 OCR,缺省截屏

Inputs: path, region.

vision.screenshotB

截屏存盘(视觉兜底/证据);路径会记住,作为 vision.diff 缺省基线

Inputs: path.

vision.vlmA

本地视觉小模型问图(crop-and-ask):缺省整屏,传 box 只裁剪该区域提问(推荐:小图快且准)。需本地 OpenAI 兼容视觉端点:设 MAH_VLM_URL 与 MAH_VLM_MODEL(如 ollama:MAH_VLM_URL=http://127.0.0.1:11434/v1/chat/completions,MAH_VLM_MODEL=qwen2.5vl:3b)。定位:图集外未知图标的语义兜底层

Inputs: box, from, prompt, timeout.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    C2.5/5.0

    Scored across 60 tools

    Disambiguation2/5

    Many tools overlap heavily across layers: ui.click vs ui.click_handle vs input.tap; ui.set_text vs ui.text_handle; ui.dump vs ui.snapshot vs ui2.state; vision.ocr vs vision.vlm vs vision.ask. Descriptions try to explain preferred/default vs fallback vs primitive, but an agent still faces multiple plausible choices for the same action.

    Naming Consistency3/5

    Most names use a readable namespace.verb_noun pattern such as ui.get_text, app.launch, vision.screenshot, but the set mixes ui vs ui2, dot-delimited namespaces, and harness-specific double-underscore names like com.android.settings__launch. The convention is largely understandable but not uniform.

    Tool Count1/5

    With 60 tools, the server is far beyond a well-scoped set and likely overwhelms tool selection. The layering of ui, ui2, input, vision, events, harness, and fallback tools suggests internal architecture leaking directly into the exposed surface.

    Completeness4/5

    The surface covers app lifecycle, UI interaction, vision/OCR, files, shell, state, events, tasks, templates, harnesses, and knowledge packages, which is very comprehensive for phone automation. Some device-level domains such as telephony, notifications, sensors, or media remain absent, but core agent workflows are well represented.

    Maintenance

    ActivitySlowing
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables MCP-compatible agents to control an Android device over the network via ADB, providing tools for shell commands, screen capture, UI inspection, file operations, and input simulation.
      4 npm
      MIT
    • F
      license
      Not graded
      quality
      C
      maintenance
      Enables AI agents and test runners to control Android devices via local ADB, including listing devices, tapping, swiping, typing, sending system keys, launching apps, and dumping UI hierarchy. Communicates over MCP stdio without exposing network listeners.
      -
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables controlling an Android device from an MCP client via ADB, including wireless debugging discovery, UI-tree inspection, and performing taps, swipes, typing, app actions, and shell commands without relying on screen coordinates.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to remotely control a real Android phone over MCP with no ADB, root, or emulator, by connecting a broker server to a sideloaded APK. Exposes tools to list devices, capture screenshots, read the UI tree, open links and deep links, and perform taps, swipes, typing, and global navigation actions.
      1
      MIT