semantic-gui-agent-testing

Semantic GUI automation for LLM agents. Drive real Android and Windows UIs through a single platform-neutral API — no hard-coded coordinates, no model calls for element localization.

Elements are located by meaning (text, content-desc, resource-id/AutomationId, natural-language description) against the accessibility tree dumped from the device, with an injectable OCR provider as a last-resort visual fallback.

Why

LLM-driven GUI agents usually fail on three things: brittle coordinate scripts, expensive screenshot-per-step vision loops, and flaky waits. This framework addresses all three:

Related MCP server: waydroid-mcp

Components

• Executors — AndroidExecutor (adb + uiautomator2) and WindowsExecutor (UI Automation via pywinauto) behind one abstract GUIExecutor.

• Semantic matcher — deterministic scoring across text / content-desc / resource-id / description surfaces, with ambiguity flagging.

• Incremental UI updates — Merkle-hash caching so unchanged screens are detected without re-processing.

• Visual fallback — pluggable OCR provider, used only when the tree fails.

• MCP server — expose the executors to any MCP-capable agent over stdio (newline-delimited JSON-RPC 2.0, no SDK dependency).

• Batch + waits + assertions — execution optimizations for agent loops.

• Sessions — per-agent isolation of executor instances and caches.

Install

Python 3.10+.

pip install -e .            # core (no device deps)
pip install -e ".[android]" # + uiautomator2
pip install -e ".[windows]" # + pywinauto (Windows only)
pip install -e ".[dev]"     # + pytest

Quickstart (no device needed)

from sgat.models import SemanticTarget
from sgat.registry import default_registry

with default_registry().create("stub") as executor:
    result = executor.tap(SemanticTarget(id="ok", text="OK"))
    print(result.success, result.detail)  # True: matched via text

Smoke-test the MCP server against the in-memory stub:

python -m sgat.mcp.server --executor stub

MCP tools

Point an MCP client at a real device:

python -m sgat.mcp.server --executor android --serial <serial>
python -m sgat.mcp.server --executor windows --window-title "Notepad"

Status

• Unit tested: 16 tests, CI on Ubuntu + Windows, Python 3.10/3.12.

• Real-device benchmarks: not yet measured. benchmarks/run_benchmarks.py is the harness; run it against your devices and record results in docs/benchmarks.md. No numbers are published here until they come from real runs.

Documentation

• AGENTS.md — implementation task brief for AI coding agents

• docs/architecture.md — design and data flow

• docs/deployment.md — Windows / Android / WSL2 setup

• docs/selection-rationale.md — technology choices and why

• docs/benchmarks.md — metrics definition and result template

License

MIT