Provides semantic GUI automation for Android devices via adb and uiautomator2, exposing MCP tools to find elements, tap, input text, navigate back, wait for elements, and assert UI state using accessibility-tree-based matching.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Semantic GUI automation for LLM agents. Drive real Android and Windows UIs through a single platform-neutral API — no hard-coded coordinates, no model calls for element localization.
Elements are located by meaning (text, content-desc, resource-id/AutomationId, natural-language description) against the accessibility tree dumped from the device, with an injectable OCR provider as a last-resort visual fallback.
LLM-driven GUI agents usually fail on three things: brittle coordinate scripts, expensive screenshot-per-step vision loops, and flaky waits. This framework addresses all three:
Problem | Approach |
Brittle coordinates | Semantic matching over the accessibility tree, deterministic and re-runnable |
Vision-loop cost | Localization is pure Python — zero model calls; OCR is only a fallback |
Flaky waits | Polling state waits, declarative assertions, per-step error-isolated batches |
Related MCP server: Deskhand-mcp
Executors — AndroidExecutor (adb + uiautomator2) and WindowsExecutor
(UI Automation via pywinauto) behind one abstract GUIExecutor.
Semantic matcher — deterministic scoring across text / content-desc / resource-id / description surfaces, with ambiguity flagging.
Incremental UI updates — Merkle-hash caching so unchanged screens are detected without re-processing.
Visual fallback — pluggable OCR provider, used only when the tree fails.
MCP server — expose the executors to any MCP-capable agent over stdio (newline-delimited JSON-RPC 2.0, no SDK dependency).
Batch + waits + assertions — execution optimizations for agent loops.
Sessions — per-agent isolation of executor instances and caches.
Python 3.10+.
pip install -e . # core (no device deps)
pip install -e ".[android]" # + uiautomator2
pip install -e ".[windows]" # + pywinauto (Windows only)
pip install -e ".[dev]" # + pytestfrom sgat.models import SemanticTarget
from sgat.registry import default_registry
with default_registry().create("stub") as executor:
result = executor.tap(SemanticTarget(id="ok", text="OK"))
print(result.success, result.detail) # True: matched via textSmoke-test the MCP server against the in-memory stub:
python -m sgat.mcp.server --executor stubTool | Purpose |
| Locate an element semantically, return best match + candidates |
| Locate and tap the element center |
| Type text, optionally into a located element |
| Platform back navigation |
| Poll until an element appears |
| Assert existence / text / enabled / clickable |
Point an MCP client at a real device:
python -m sgat.mcp.server --executor android --serial <serial>
python -m sgat.mcp.server --executor windows --window-title "Notepad"Unit tested: 16 tests, CI on Ubuntu + Windows, Python 3.10/3.12.
Real-device benchmarks: not yet measured. benchmarks/run_benchmarks.py
is the harness; run it against your devices and record results in
docs/benchmarks.md. No numbers are published here
until they come from real runs.
AGENTS.md — implementation task brief for AI coding agents
docs/architecture.md — design and data flow
docs/deployment.md — Windows / Android / WSL2 setup
docs/selection-rationale.md — technology choices and why
docs/benchmarks.md — metrics definition and result template
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Let AI agents control Android and iOS devices on Mobilerun Cloud: screen, input, apps, workflows.
Phones for your agents: each agent gets its own cloud Android phone over MCP, reads the screen, taps, types and installs your APK. Ready in ~20 s, $0.01/min, parked phones free.
Give an AI agent an iPhone to see, tap and type on over MCP. Free early access (Simulator).
Automate 1,000+ services from any MCP-compatible AI agent: build Applets, run actions and queries.
