cua
by Jayanyixing
README.md
# computer-use-agy
macOS desktop Computer Use plugin & MCP server for **Antigravity** (Skill + `cua` stdio MCP server + Lifecycle Hooks).
Inspired by the Codex Computer Use interaction loop (`observe` → `batch act` → `verify`), implemented directly on top of macOS public frameworks (`Quartz`, `Cocoa`, `ApplicationServices`, `Pillow`) without relying on private IPC helpers.
## Features
- **Background Window Capture (Zero-Focus-Steal when full-sized)**:
- `screenshot(app?, max_width?, restore_focus?)`: Captures an individual background app window via `CGWindowListCopyWindowInfo` + `screencapture -l` without stealing focus when full-sized; automatically handles Stage Manager strip thumbnails with optional `restore_focus=true`.
- **Deep Accessibility Inspection & Semantic Targeting**:
- `ax_snapshot`: Reads background apps without stealing focus (`max_depth=35`, skips `AXMenuBar` and `0x0` hidden nodes by default so React/Next.js web apps fit cleanly).
- `click(index | label+role | x,y)`: Supports cached `index`, live semantic AX matching (`app` + `label`/`role`/`value`), and point coordinates with target-app frontmost guard.
- `wait_for(app, label?, role?, value?, timeout_ms?)`: Polls the live AX tree for dynamic UI changes.
- `set_value(text, app?, index?|label?)`: Sets `AXValue` directly when supported (background-safe), falling back cleanly to `click` + `Cmd+A` + `type_text`.
- **Single-Turn Batched Automation & Lifecycle Hooks**:
- `batch`: Chains `focus_app`, `click`, `drag`, `scroll`, `type_text`, `press_key`, `move_mouse`, `wait`, `wait_for`, `set_value`, and `ax_snapshot` in one MCP turn, returning the final screenshot and optionally restoring focus (`restore_focus=true`).
- `hooks.json` + [hooks/cua_hooks.py](./hooks/cua_hooks.py): Tracks turn-level focus changes on `PostToolUse` and cleans up / restores Antigravity focus on `Stop` if a CUA flow is canceled or errors mid-flight.
## Installation
### Requirements
- macOS (tested on Apple Silicon / macOS 27.3)
- Python 3.11+ (automatically provisioned if [`uv`](https://docs.astral.sh/uv/) is installed)
### Install into Antigravity
```bash
git clone https://github.com/Jayanyixing/computer-use-agy.git
cd computer-use-agy
bash scripts/install.sh
```
`scripts/install.sh` sets up `.venv` and links the repository into `~/.gemini/config/plugins/computer-use` (pass `--copy` if you prefer copying instead of symlinking).
### Grant macOS Permissions
In **System Settings → Privacy & Security**:
1. **Screen Recording (`录屏与系统录音`)**: Add and enable **Antigravity**.
2. **Accessibility (`辅助功能`) / Device Control & Data Access (`设备控制和数据访问` on newer macOS)**: Add and enable **Antigravity**.
> **Important**: Fully quit (`Cmd+Q`) and reopen Antigravity after granting permissions so MCP child processes inherit them.
## Structure
```text
computer-use-agy/
├── plugin.json # Antigravity plugin manifest
├── mcp_config.json # Plugin-scoped MCP server registration
├── hooks.json # PostToolUse & Stop lifecycle hooks
├── pyproject.toml # Python project metadata & dependencies
├── requirements.txt # pip requirements
├── assets/logo.svg # Plugin icon
├── hooks/cua_hooks.py # Turn focus state cleanup hook
├── server/cua_mcp.py # macOS stdio MCP server (14 tools)
├── scripts/
│ ├── install.sh # Venv setup + Antigravity plugin installer
│ ├── run-hook.sh # Portable hook runner
│ └── start-mcp.sh # Portable MCP launcher
└── skills/computer-use/SKILL.md # Agent workflow & safety skill
```
## License
[MIT](./LICENSE)
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues