Skip to main content
Glama
README.md
# computer-use-agy

macOS desktop Computer Use plugin & MCP server for **Antigravity** (Skill + `cua` stdio MCP server + Lifecycle Hooks).

Inspired by the Codex Computer Use interaction loop (`observe` → `batch act` → `verify`), implemented directly on top of macOS public frameworks (`Quartz`, `Cocoa`, `ApplicationServices`, `Pillow`) without relying on private IPC helpers.

## Features

- **Background Window Capture (Zero-Focus-Steal when full-sized)**:
  - `screenshot(app?, max_width?, restore_focus?)`: Captures an individual background app window via `CGWindowListCopyWindowInfo` + `screencapture -l` without stealing focus when full-sized; automatically handles Stage Manager strip thumbnails with optional `restore_focus=true`.
- **Deep Accessibility Inspection & Semantic Targeting**:
  - `ax_snapshot`: Reads background apps without stealing focus (`max_depth=35`, skips `AXMenuBar` and `0x0` hidden nodes by default so React/Next.js web apps fit cleanly).
  - `click(index | label+role | x,y)`: Supports cached `index`, live semantic AX matching (`app` + `label`/`role`/`value`), and point coordinates with target-app frontmost guard.
  - `wait_for(app, label?, role?, value?, timeout_ms?)`: Polls the live AX tree for dynamic UI changes.
  - `set_value(text, app?, index?|label?)`: Sets `AXValue` directly when supported (background-safe), falling back cleanly to `click` + `Cmd+A` + `type_text`.
- **Single-Turn Batched Automation & Lifecycle Hooks**:
  - `batch`: Chains `focus_app`, `click`, `drag`, `scroll`, `type_text`, `press_key`, `move_mouse`, `wait`, `wait_for`, `set_value`, and `ax_snapshot` in one MCP turn, returning the final screenshot and optionally restoring focus (`restore_focus=true`).
  - `hooks.json` + [hooks/cua_hooks.py](./hooks/cua_hooks.py): Tracks turn-level focus changes on `PostToolUse` and cleans up / restores Antigravity focus on `Stop` if a CUA flow is canceled or errors mid-flight.

## Installation

### Requirements

- macOS (tested on Apple Silicon / macOS 27.3)
- Python 3.11+ (automatically provisioned if [`uv`](https://docs.astral.sh/uv/) is installed)

### Install into Antigravity

```bash
git clone https://github.com/Jayanyixing/computer-use-agy.git
cd computer-use-agy
bash scripts/install.sh
```

`scripts/install.sh` sets up `.venv` and links the repository into `~/.gemini/config/plugins/computer-use` (pass `--copy` if you prefer copying instead of symlinking).

### Grant macOS Permissions

In **System Settings → Privacy & Security**:

1. **Screen Recording (`录屏与系统录音`)**: Add and enable **Antigravity**.
2. **Accessibility (`辅助功能`) / Device Control & Data Access (`设备控制和数据访问` on newer macOS)**: Add and enable **Antigravity**.

> **Important**: Fully quit (`Cmd+Q`) and reopen Antigravity after granting permissions so MCP child processes inherit them.

## Structure

```text
computer-use-agy/
├── plugin.json                      # Antigravity plugin manifest
├── mcp_config.json                  # Plugin-scoped MCP server registration
├── hooks.json                       # PostToolUse & Stop lifecycle hooks
├── pyproject.toml                   # Python project metadata & dependencies
├── requirements.txt                 # pip requirements
├── assets/logo.svg                  # Plugin icon
├── hooks/cua_hooks.py               # Turn focus state cleanup hook
├── server/cua_mcp.py                # macOS stdio MCP server (14 tools)
├── scripts/
│   ├── install.sh                   # Venv setup + Antigravity plugin installer
│   ├── run-hook.sh                  # Portable hook runner
│   └── start-mcp.sh                 # Portable MCP launcher
└── skills/computer-use/SKILL.md     # Agent workflow & safety skill
```

## License

[MIT](./LICENSE)