# twiddle-mcp

> Stop your agent twiddling its thumbs. — a [Twiddle Thumb Studio](https://github.com/roti-c137) tool

**Blocking `wait_for` primitives and semantic UI diffs for the iOS Simulator and Android Emulator.** An MCP server that lets coding agents (Claude Code, Cursor, any MCP client) drive and — critically — *verify* mobile apps without screenshot → sleep → screenshot polling.

Web agents already have this: Playwright MCP's accessibility snapshots and `browser_wait_for`. On mobile, waiting exists only in fragments — Maestro's flow engine has implicit waits, SilbercueSwift waits on log patterns (iOS only), Appium-based stacks can script visibility waits — but no cross-platform MCP server makes blocking tree-based waits and semantic diffs the primitives agents call directly. This fills that gap.

```
agent: tap(ref=e4)                      → "Tapped button 'Load Data'"
agent: wait_for({text: "Welcome back"}) → blocks…
                                        → MATCHED in 1452ms:
                                          ~ text #statusLabel "Loading…" → "Welcome back, Tyler"
```

One tool call. No sleeps, no screenshot diffing, no wasted tokens.

## Setup

```json
// .mcp.json
{
  "mcpServers": {
    "twiddle": { "command": "npx", "args": ["-y", "@twiddlethumb/twiddle-mcp"] }
  }
}
```

Requirements: macOS with Xcode (iOS) and/or the Android SDK with `adb` on PATH. First iOS use compiles WebDriverAgent (~30s–3min, then cached). Android bootstraps in ~3s.

## Tools

**Verify primitives (the point of this project):**

| Tool | What it does |
|---|---|
| `wait_for` | Block until text/element appears (or `state:"gone"`). Returns elapsed ms + semantic diff. On timeout returns `matched:false` **with the current tree** — never throws. |
| `wait_for_idle` | Block until the UI stops changing. Volatile elements (clocks, spinners) are auto-detected and ignored. |
| `get_ui_tree` | Accessibility tree as text with `[ref=eN]` markers — cheaper and more precise than a screenshot. |
| `ui_diff` | "What changed since snapshot N?" as short sentences. |
| `handle_alert` | Accept/dismiss system & in-app dialogs from the tree — handles entrance-animation frames and the tap-jacking cooldown that silently swallows an agent's too-fast taps, with verify-and-retry. |

**Driver:** `list_devices` · `launch_app` · `terminate_app` · `tap` (by ref / selector / x,y — refs re-resolve against the live tree; a vanished element returns the current tree instead of a blind tap; `scroll:true` swipes through content until the selector appears, settle-aware so it never taps a still-moving list) · `type_text` · `swipe` · `press_button` · `screenshot` (fallback for purely visual checks).

## How it works

Both platforms expose a localhost HTTP automation server — WebDriverAgent on iOS (bootstrapped from the `appium-webdriveragent` npm package), `appium-uiautomator2-server` on Android (APKs installed over adb). One keep-alive connection pool and **one shared poll loop per device** (~7Hz) serve any number of concurrent waits. Trees are normalized to a platform-agnostic node model; elements get stable refs via identity keys (`role+identifier` first), so refs survive re-renders; diffs align by those keys and suppress frame jitter and volatile elements.

## Development

```sh
pnpm install
pnpm test        # unit tests (tree engine, wait logic) — no devices needed
pnpm build       # dist/index.js
examples/ios-demo/build.sh   # build + install the SwiftUI demo on the booted sim
```

## Benchmark

Measured against mobile-mcp, Maestro MCP, and the raw screenshot baseline on both platforms (tap → verify a 2s async load; medians over repeated runs, canonical agent patterns, full method in [bench/RESULTS.md](bench/RESULTS.md)):

| | agent turns | context/verification | est. total (Android / iOS) |
|---|---|---|---|
| **twiddle-mcp** | **2** | **1 KB** | **11s / 9s** |
| mobile-mcp | 3–4 (1 poll/turn) | 3 KB | 16s / 14s* |
| maestro-mcp | 1 (flow YAML) | 0.3 KB | 10s / 6s |
| screenshots + shell (default) | 3 | **117 KB** | 12s / ✗ (simctl cannot tap) |

\* iOS requires a workaround for mobile-mcp's fractional-coordinate click bug (see RESULTS.md).

On failure (`wait_for` timeout), the agent gets the full current UI tree to reason over; a failed Maestro flow returns a 288-byte error string, and polling patterns return their last snapshot. Every extra second of app latency costs polling tools another turn; it costs `wait_for` nothing.

## Status

Early but working: iOS + Android verified end-to-end over real MCP stdio (see `bench/` and `examples/`). Roadmap: event-driven AccessibilityService transport on Android, iOS benchmark, agent-level benchmark via headless Claude Code sessions (`bench/run.mjs`).

MIT