macos-ui-mcp
Inspect and drive a debug app's UI through MCP tools, without Screen Recording permission.
List the app's on-screen windows, including id, title, key status, and frame.
Dump the semantic UI tree as JSON or a compact outline, optionally for a specific window.
Render a window or a selected node to PNG in-process.
Click a view matched by selector.
Set text in a field matched by selector.
Invoke a named app-registered action hook.
Poll the UI tree until a selector appears or a timeout expires.
Note: the schema does not expose README-mentioned launch_app, tap_at, or scroll tools.
Provides tools to inspect and drive native macOS app UI, including listing windows, getting UI trees, taking screenshots, clicking, typing, waiting for elements, and invoking actions, without requiring Screen Recording permission.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@macos-ui-mcp Inspect the UI tree, find the Save button, click it, and screenshot the result."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
macos-ui-mcp
A Playwright-style tool that lets Claude inspect and drive a native macOS app's UI while you build it — without Screen Recording permission. The same harness runs in iOS Simulator apps, including React Native / Expo apps.
Instead of capturing the screen, the app renders itself. A tiny debug-only harness inside your app exposes its view tree and an in-process PNG renderer over localhost HTTP; an MCP server turns that into tools Claude can call.
See IDEA.md for the design rationale, the architecture comparison, and the project goals.
Architecture
flowchart LR
Claude["Claude Code"] -- "MCP (stdio)" --> Server["macos-ui-mcp server\n(src/)"]
Server -- "HTTP :8787" --> Harness["AppMCP harness\n(DEBUG builds only)"]
subgraph App["Your app process"]
Harness
endThe server and the harness talk one shared HTTP contract
(src/contract.ts), so either harness implementation works with
the same MCP tools:
flowchart TB
Contract["HTTP contract\nGET /windows · /tree · /snapshot\nPOST /tap · /setText · /action"]
Contract --> Swift["swift-harness/\nAppKit / UIKit / SwiftUI — NSView / UIView tree"]
Contract --> Tauri["tauri-harness/\nTauri v2 — webview DOM"]Related MCP server: Desktop Pilot MCP
Tools
Tool | Does |
| Launch the target app ( |
| List the app's on-screen windows (id, title, key, frame). |
| Dump the semantic view tree: roles, labels, values, enabled/focused state, frames, stable ids. |
| Render a window — or a single node via |
| Tap a view matched by selector. On iOS, non- |
| Set the text value of a field matched by selector. On iOS it types like the keyboard (delegate + |
| Tap at a window coordinate with a real synthesized touch (iOS). |
| Scroll a scroll view with a synthesized drag, no momentum (iOS). |
| Trigger a named action hook the app registered via |
| Poll the UI tree until a selector appears, or time out. |
A typical session — Claude launches the app itself, then drives it:
sequenceDiagram
participant C as Claude
participant M as MCP server
participant H as Harness (in app)
C->>M: launch_app(path)
M->>H: spawn app, poll GET /windows
H-->>M: 200 { windows: [...] }
C->>M: get_ui_tree()
M->>H: GET /tree
H-->>M: view tree JSON
C->>M: click(selector)
M->>H: POST /tap
H-->>M: { ok: true }
C->>M: screenshot()
M->>H: GET /snapshot
H-->>M: PNG bytesLayout
Path | What |
| the MCP server (TypeScript) |
| the harness HTTP wire format (types + zod schemas) |
| vitest suite + an in-memory mock harness; |
| drop-in harness for native AppKit / SwiftUI apps |
| drop-in harness (Tauri v2 plugin) for webview apps |
| a minimal AppKit app that links the harness — used by the e2e test |
| a UIKit app shaped like React Native — used by the iOS e2e test |
| Expo config plugin that embeds the harness in Debug iOS builds |
| a minimal Expo / React Native app using the plugin |
Use it
1. Add the harness to your app (DEBUG builds only)
AppKit / SwiftUI — see swift-harness/README.md. One line
in your AppDelegate:
#if DEBUG
AppMCP.start() // opens http://127.0.0.1:8787
#endifTauri v2 — see tauri-harness/README.md. Register the
plugin behind #[cfg(debug_assertions)] and add "macos-ui-mcp:default" to your
capabilities:
#[cfg(debug_assertions)]
{
builder = builder.plugin(tauri_plugin_macos_ui_mcp::init());
}Expo / React Native — see expo-plugin/README.md. Add the
plugin to app.config.js; expo prebuild links the harness (Debug only) and starts it.
2. Build the MCP server
npm install
npm run build3. Register it with Claude Code
// .mcp.json
{
"mcpServers": {
"macos-ui-mcp": {
"command": "node",
"args": ["/absolute/path/to/macos-ui-mcp/dist/index.js"],
"env": { "MACOS_UI_MCP_HARNESS_URL": "http://127.0.0.1:8787" }
}
}
}iOS Simulator
The same swift-harness builds for iOS (UIKit). Add it to your iOS app, build for a
simulator (Debug-iphonesimulator), boot a simulator, then:
launch_app(path: ".../Debug-iphonesimulator/MyApp.app", simulator: "booted")This runs simctl install + simctl launch --terminate-running-process (bundle id read
from Info.plist; env is forwarded as SIMCTL_CHILD_*). Simulator apps share the
Mac's loopback, so the harness is still reached at http://127.0.0.1:8787.
Now ask Claude to call launch_app (path to your .app bundle or executable) to start
your debug build and wait for the harness to come up, then get_ui_tree, click,
type, or screenshot — no Screen Recording prompt.
Selector grammar
Used by click, type, screenshot, wait_for:
Form | Matches |
| node whose id is |
| role |
| any node with label |
| any node with role |
"Label" also matches visible text (React Native <Text>), and on iOS only on-screen
nodes are listed: hidden, transparent and scrolled-away views are left out.
When several nodes match, a match nested inside another match counts as one element (an RN
Pressable and the Text inside it), and nodes marked hittable: false (covered, e.g. by
the next screen) are dropped. If more than one match is still left, click / type
refuse and report the candidate ids.
Develop
npm test # unit tests, no real app needed (mock harness); e2e is skipped
npm run typecheck
npm run dev # run the server against a live harness
npm run test:e2e # builds showcase/, launches it, runs the real tools against the
# real Swift harness. Needs a logged-in macOS GUI session.
npm run test:e2e:ios # builds showcase-ios/, installs it on the booted simulator, drives it
npm run test:e2e:rn # drives examples/expo-rn (real RN app; build it first, see its README)npm run test:e2e is the proof of the goals: it asserts get_ui_tree sees the
controls, screenshot returns a real in-process PNG, and click / type change the
live UI — with no Screen Recording permission.
Status
AppKit / SwiftUI (
swift-harness/) — all 7 harness-specific tools. 56 unit testsa 5-case end-to-end test driving the
showcase/app through the real harness. Verified: tree, in-process PNG snapshot, tap, set-text, action hooks — Screen Recording off.
Tauri v2 (
tauri-harness/) — plugin compiles (cargo check/clippyclean); full DOM tree, DOM-event interaction, dependency-free snapshot. Not yet exercised by an automated end-to-end test in this repo (no Tauri showcase app).iOS / iOS Simulator (
swift-harness/, UIKit) — tree with visibility filtering andhittable, in-process snapshot, real synthesized touches (click,tap_at,scroll), keyboard-path typing. Verified on the iOS 26.2 simulator bytest:e2e:ios(RN-shaped UIKit app) andtest:e2e:rn(real Expo 57 / RN 0.86 app).Expo / React Native (
expo-plugin/) — Debug-only pod +AppMCP.starton everyexpo prebuild; verified on a real prebuild (Release links nothing).launch_app— framework-agnostic (spawns/opens whatever path you give it, then polls the harness); withsimulator,simctl install+launch. Used by both iOS e2e suites.Not yet — SwiftUI-native tree (AppKit introspection only for now), Windows adapter, third-party-app inspection via the Accessibility API.
License
MIT © 2026 Thang Duong
Available Tools
7 toolsclickClickB
Tap a view matched by selector. Selector grammar: "#id" | Role "Label" | "Label" | Role.
Inputs: selector (required), window.
get_ui_treeGet UI treeA
Dump the semantic view tree of a window as JSON: roles, labels, values, enabled/focused state, frames, and stable ids. Use compact for a terse outline.
Inputs: window, compact.
invoke_actionInvoke actionC
Trigger a named action hook registered by the app via AppMCP.registerAction(name, …).
Inputs: name (required).
list_windowsList windowsA
List the app's on-screen windows (id, title, key, frame).
Inputs: no named parameters.
screenshotScreenshotA
Render a window (or a single node via selector) to PNG. Rendered in-process by the app — no Screen Recording permission.
Inputs: window, selector.
typeTypeC
Set the text value of a field matched by selector.
Inputs: text (required), selector (required), window.
wait_forWait forB
Poll the UI tree until a selector appears, or time out.
Inputs: selector (required), pollMs, window, timeoutMs.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 7 tools
Each tool has a clearly distinct purpose: inspection (get_ui_tree, list_windows, screenshot), interaction (click, type, invoke_action), and synchronization (wait_for). No overlapping functionality; an agent can unambiguously select the right tool for a task.
Most tools follow a verb-oriented pattern (get_ui_tree, list_windows, invoke_action, wait_for), while click and type are single verbs and screenshot is a noun used as a verb. The pattern is generally predictable, with only minor deviations from a strict verb_noun structure.
Seven tools is well-scoped for a UI automation server. Each tool earns its place—reading state, capturing visuals, performing actions, and waiting—without unnecessary bloat or an overly sparse surface.
The surface covers the core UI workflow (inspect, interact, wait), but misses common operations like scrolling or keyboard shortcuts. These are not critical for many automation scenarios, so the set is mostly complete with minor gaps.
Maintenance
Related MCP Connectors
Drive real devices from your AI Coding tool. Embed a client SDK (Unity, Godot, Flutter, iOS/macOS, Android, React Native, Web) in your app, then capture screenshots, traverse the UI tree, inject taps and key events, and run automated test tasks on the physical device over a secure relay.
Cloud iPhones and Macs with Xcode for AI agents: build, run and test iOS apps, use the screen.
The cloud Mac alone, with Xcode, for AI agents: run commands, build and test, use the screen.
Let your AI agent control real iPhones: screenshots, tap, swipe, type and open apps.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables Claude to see and interact with any macOS application using natural language commands. Perfect for testing Mac applications, UI automation, and app development with AI assistance.34MIT
- AlicenseNot gradedqualityDmaintenanceEnables high-speed native macOS automation for Claude by interacting directly with the Accessibility API, AppleScript, and UI trees instead of using screenshots. It allows users to read app states, click elements, and type text semantically across any macOS application.11 npm13MIT
- AlicenseAqualityBmaintenanceMCP server that lets Claude use your Mac — native apps and the browser — by remembering what it has seen instead of screenshotting every step.122MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI assistants to inspect and control macOS apps via accessibility trees, screenshots, OCR, and input simulation, with a visible pointer.MIT