cua
Provides tools for controlling the macOS desktop: capturing background app windows without stealing focus, reading app accessibility trees for semantic targeting, and acting via clicks, drags, scrolling, typing, key presses, and value setting, plus batched single-turn automation with verification screenshots.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
computer-use-agy
macOS desktop Computer Use plugin & MCP server for Antigravity (Skill + cua stdio MCP server + Lifecycle Hooks).
Inspired by the Codex Computer Use interaction loop (observe → batch act → verify), implemented directly on top of macOS public frameworks (Quartz, Cocoa, ApplicationServices, Pillow) without relying on private IPC helpers.
Features
Background Window Capture (Zero-Focus-Steal when full-sized):
screenshot(app?, max_width?, restore_focus?): Captures an individual background app window viaCGWindowListCopyWindowInfo+screencapture -lwithout stealing focus when full-sized; automatically handles Stage Manager strip thumbnails with optionalrestore_focus=true.
Deep Accessibility Inspection & Semantic Targeting:
ax_snapshot: Reads background apps without stealing focus (max_depth=35, skipsAXMenuBarand0x0hidden nodes by default so React/Next.js web apps fit cleanly).click(index | label+role | x,y): Supports cachedindex, live semantic AX matching (app+label/role/value), and point coordinates with target-app frontmost guard.wait_for(app, label?, role?, value?, timeout_ms?): Polls the live AX tree for dynamic UI changes.set_value(text, app?, index?|label?): SetsAXValuedirectly when supported (background-safe), falling back cleanly toclick+Cmd+A+type_text.
Single-Turn Batched Automation & Lifecycle Hooks:
batch: Chainsfocus_app,click,drag,scroll,type_text,press_key,move_mouse,wait,wait_for,set_value, andax_snapshotin one MCP turn, returning the final screenshot and optionally restoring focus (restore_focus=true).hooks.json+ hooks/cua_hooks.py: Tracks turn-level focus changes onPostToolUseand cleans up / restores Antigravity focus onStopif a CUA flow is canceled or errors mid-flight.
Related MCP server: macos-desktop-control
Installation
Requirements
macOS (tested on Apple Silicon / macOS 27.3)
Python 3.11+ (automatically provisioned if
uvis installed)
Install into Antigravity
git clone https://github.com/Jayanyixing/computer-use-agy.git
cd computer-use-agy
bash scripts/install.shscripts/install.sh sets up .venv and links the repository into ~/.gemini/config/plugins/computer-use (pass --copy if you prefer copying instead of symlinking).
Grant macOS Permissions
In System Settings → Privacy & Security:
Screen Recording (
录屏与系统录音): Add and enable Antigravity.Accessibility (
辅助功能) / Device Control & Data Access (设备控制和数据访问on newer macOS): Add and enable Antigravity.
Important: Fully quit (
Cmd+Q) and reopen Antigravity after granting permissions so MCP child processes inherit them.
Structure
computer-use-agy/
├── plugin.json # Antigravity plugin manifest
├── mcp_config.json # Plugin-scoped MCP server registration
├── hooks.json # PostToolUse & Stop lifecycle hooks
├── pyproject.toml # Python project metadata & dependencies
├── requirements.txt # pip requirements
├── assets/logo.svg # Plugin icon
├── hooks/cua_hooks.py # Turn focus state cleanup hook
├── server/cua_mcp.py # macOS stdio MCP server (14 tools)
├── scripts/
│ ├── install.sh # Venv setup + Antigravity plugin installer
│ ├── run-hook.sh # Portable hook runner
│ └── start-mcp.sh # Portable MCP launcher
└── skills/computer-use/SKILL.md # Agent workflow & safety skillLicense
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Securely control computers you explicitly pair through files, terminals, processes, screenshots, desktop UI/input, clipboard, browser automation, diagnostics, and document tools.
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
- DNS-DATA RemoteDeprecatedcom.dns-data
Connect AI assistants to authorized Windows and macOS devices for files, commands, and management.
Build, run, inspect, and continue AgentScript workflows on connected Super Chrome and Mac clients.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables AI assistants to automate macOS desktop tasks including mouse control, keyboard input, screenshots, window management, and UI interaction.417MIT
- AlicenseNot gradedqualityDmaintenanceProvides native macOS desktop automation for AI agents, enabling screen capture, mouse/keyboard control, window management, and iOS/Android simulator control in both foreground and background modes without focus stealing.6MIT
- FlicenseNot gradedqualityBmaintenanceEnables macOS computer-use automation through public Apple APIs, supporting app and window discovery, clicking, typing, scrolling, and field setting via accessibility controls.5-
- FlicenseNot gradedqualityCmaintenanceEnables macOS GUI automation via screenshots, mouse/keyboard control, accessibility tree inspection, and app/window management.1-