Skip to main content
Glama

computer-use-agy

macOS desktop Computer Use plugin & MCP server for Antigravity (Skill + cua stdio MCP server + Lifecycle Hooks).

Inspired by the Codex Computer Use interaction loop (observe → batch act → verify), implemented directly on top of macOS public frameworks (Quartz, Cocoa, ApplicationServices, Pillow) without relying on private IPC helpers.

Features

  • Background Window Capture (Zero-Focus-Steal when full-sized):

    • screenshot(app?, max_width?, restore_focus?): Captures an individual background app window via CGWindowListCopyWindowInfo + screencapture -l without stealing focus when full-sized; automatically handles Stage Manager strip thumbnails with optional restore_focus=true.

  • Deep Accessibility Inspection & Semantic Targeting:

    • ax_snapshot: Reads background apps without stealing focus (max_depth=35, skips AXMenuBar and 0x0 hidden nodes by default so React/Next.js web apps fit cleanly).

    • click(index | label+role | x,y): Supports cached index, live semantic AX matching (app + label/role/value), and point coordinates with target-app frontmost guard.

    • wait_for(app, label?, role?, value?, timeout_ms?): Polls the live AX tree for dynamic UI changes.

    • set_value(text, app?, index?|label?): Sets AXValue directly when supported (background-safe), falling back cleanly to click + Cmd+A + type_text.

  • Single-Turn Batched Automation & Lifecycle Hooks:

    • batch: Chains focus_app, click, drag, scroll, type_text, press_key, move_mouse, wait, wait_for, set_value, and ax_snapshot in one MCP turn, returning the final screenshot and optionally restoring focus (restore_focus=true).

    • hooks.json + hooks/cua_hooks.py: Tracks turn-level focus changes on PostToolUse and cleans up / restores Antigravity focus on Stop if a CUA flow is canceled or errors mid-flight.

Related MCP server: macos-desktop-control

Installation

Requirements

  • macOS (tested on Apple Silicon / macOS 27.3)

  • Python 3.11+ (automatically provisioned if uv is installed)

Install into Antigravity

git clone https://github.com/Jayanyixing/computer-use-agy.git
cd computer-use-agy
bash scripts/install.sh

scripts/install.sh sets up .venv and links the repository into ~/.gemini/config/plugins/computer-use (pass --copy if you prefer copying instead of symlinking).

Grant macOS Permissions

In System Settings → Privacy & Security:

  1. Screen Recording (录屏与系统录音): Add and enable Antigravity.

  2. Accessibility (辅助功能) / Device Control & Data Access (设备控制和数据访问 on newer macOS): Add and enable Antigravity.

Important: Fully quit (Cmd+Q) and reopen Antigravity after granting permissions so MCP child processes inherit them.

Structure

computer-use-agy/
├── plugin.json                      # Antigravity plugin manifest
├── mcp_config.json                  # Plugin-scoped MCP server registration
├── hooks.json                       # PostToolUse & Stop lifecycle hooks
├── pyproject.toml                   # Python project metadata & dependencies
├── requirements.txt                 # pip requirements
├── assets/logo.svg                  # Plugin icon
├── hooks/cua_hooks.py               # Turn focus state cleanup hook
├── server/cua_mcp.py                # macOS stdio MCP server (14 tools)
├── scripts/
│   ├── install.sh                   # Venv setup + Antigravity plugin installer
│   ├── run-hook.sh                  # Portable hook runner
│   └── start-mcp.sh                 # Portable MCP launcher
└── skills/computer-use/SKILL.md     # Agent workflow & safety skill

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI assistants to automate macOS desktop tasks including mouse control, keyboard input, screenshots, window management, and UI interaction.
      417
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Provides native macOS desktop automation for AI agents, enabling screen capture, mouse/keyboard control, window management, and iOS/Android simulator control in both foreground and background modes without focus stealing.
      6
      MIT
    • F
      license
      Not graded
      quality
      B
      maintenance
      Enables macOS computer-use automation through public Apple APIs, supporting app and window discovery, clicking, typing, scrolling, and field setting via accessibility controls.
      5
      -