Skip to main content
Glama

ApexUse — Computer Use MCP Server for AI Agents (macOS, Windows & Linux)

CI Crates.io License: MIT Rust Zero Dependencies MCP

ApexUse (apexuse-mcp) is an open-source, local-first Computer Use MCP server written in pure Rust and native OS bridges. It gives AI coding agents — including Claude Code, OpenAI Codex, Gemini CLI, and Antigravity — full desktop and browser control over macOS, Windows, and Linux using native Accessibility trees, Set-of-Mark ([m1]..[mN]) visual grounding, local OCR, a non-hijacking AgentCursor overlay, batched run_actions, and local workflow memory.

Zero external LLM keys. Zero external Rust crate dependencies. Zero cloud relay.


Why ApexUse vs. Alternative Computer-Use Tools?

Capability

ApexUse (apexuse-mcp)

Screenshot-Only Computer Use

Browser-Only MCPs (Playwright / DevTools)

App Coverage

Full OS: native apps, Electron, terminals & any browser

Whole screen

Browser tabs only

UI Grounding

Hybrid: semantic Accessibility (e1..eN) + Set-of-Mark ([m1]..[mN]) + OCR

Pixel coordinates only

DOM only

Multi-Step Speed

run_actions (up to 12 local chained steps) + local wait_for polling

1 LLM round-trip per click

Step-by-step tool calls

Scroll & Verification

Target-scoped scroll (ref/mark/x,y) + verify_change auto-retry

Blind scroll + manual re-screenshot

Page scroll only

Visual Pointer

Floating transparent AgentCursor overlay (never warps macOS hardware cursor)

Teleports real user cursor

None / page focus

Workflow Memory

Records & replays structural macros via workflow_memory

Re-plans every run

Scripts must be coded

Security Controls

Native password/secure-field blocking, protected-app list, STOP kill switch, audit log

Sends full raw screen pixels

Web origin scope


Related MCP server: oswright

Platform Support

Platform

Backend Driver

Capabilities & Verification

macOS 14+

MacosDriver + rust/native/macos_bridge.m

AXUIElement, ScreenCaptureKit, Apple Vision OCR, ApexUseCursorView overlay, per-PID CGEvent/NSEvent. Live verified on macOS.

Windows 10 / 11

WindowsDriver (rust/src/driver/windows.rs)

Pure-Rust Win32 FFI (IUIAutomation COM, PrintWindow/BitBlt, WinRT Windows.Media.Ocr, SendInput, layered AgentCursor, som_png overlay). Cross-checked (x86_64-pc-windows-gnu) & built in Windows CI; live Windows host test pending.

Linux (X11 & Wayland)

LinuxDriver (rust/src/driver/linux.rs)

Runtime dlopen libX11 / libXtst + AT-SPI2 (gi.repository.Atspi), Wayland helpers (grim / spectacle / gnome-screenshot, wtype / ydotool), tesseract OCR, som_png overlay. Cross-checked (x86_64-unknown-linux-gnu) & built in Linux CI; live Linux host test pending.


Quick Start

1. Install or build from source

Download prebuilt binaries for macOS (arm64), Linux (x86_64), or Windows (x86_64) from GitHub Releases, or install directly from crates.io:

cargo install apexuse-mcp

Or clone and build directly (requires only the Rust toolchain, plus Xcode Command Line Tools on macOS):

git clone https://github.com/Jackpkn/apexuse.git
cd apexuse
cargo build --release --manifest-path rust/Cargo.toml

Output binary: rust/target/release/apexuse-mcp (or apexuse-mcp.exe on Windows).

2. Check OS readiness & permissions

./rust/target/release/apexuse-mcp doctor
  • macOS: Enable Accessibility and Screen Recording for the terminal or IDE launching apexuse-mcp in System Settings → Privacy & Security.

  • Windows: Uses built-in Win32 / UI Automation / WinRT APIs with no extra dependencies.

  • Linux: Runs on X11 via libX11/libXtst (dlopen). For full semantic accessibility, OCR, and Wayland support install distro packages as needed (e.g., python3-gi, gir1.2-atspi-2.0, tesseract-ocr, xdotool, or grim/wtype/ydotool on Wayland).

3. Connect to your AI coding agent

Generate a ready-to-paste snippet for your client:

./rust/target/release/apexuse-mcp install antigravity
# Supported: antigravity | gemini | claude | codex
  • Claude Code:

    claude mcp add apexuse --env APEXUSE_POLICY=/abs/path/to/apexuse/policy.autonomous.yaml -- /abs/path/to/apexuse/rust/target/release/apexuse-mcp
  • Antigravity / Gemini CLI (~/.gemini/antigravity/mcp_config.json or ~/.gemini/settings.json):

    {
      "mcpServers": {
        "apexuse": {
          "command": "/abs/path/to/apexuse/rust/target/release/apexuse-mcp",
          "args": [],
          "env": {
            "APEXUSE_POLICY": "/abs/path/to/apexuse/policy.autonomous.yaml"
          }
        }
      }
    }
  • OpenAI Codex (~/.codex/config.toml):

    [mcp_servers.apexuse]
    command = "/abs/path/to/apexuse/rust/target/release/apexuse-mcp"
    args = []
    startup_timeout_sec = 60
    tool_timeout_sec = 180
    
    [mcp_servers.apexuse.env]
    APEXUSE_POLICY = "/abs/path/to/apexuse/policy.autonomous.yaml"

See agent-configs/README.md and the bundled skill at .agents/skills/apexuse/SKILL.md.


MCP Tools Reference (6 Tools)

Tool

Actions & Key Parameters

inspect_app

Read-only Accessibility snapshot for app (bundle ID, executable, app name, or "desktop"), optional window index and query filter. Returns e1..eN refs & workflow hints without taking a screenshot.

visual_inspect

Captures an app window or full "desktop", fuses Accessibility + OCR (ocr="auto"|"fast"|"accurate"|"off", max_marks 1..=150), and returns an annotated PNG with [m1]..[mN] mark IDs.

computer

inspect, click, double_click, right_click, hover (ref/mark/x,y), drag, type, press_key, scroll (optional ref/mark/x,y target), run_actions (up to 12 batched steps + inspect_after), wait_for (local AX/OCR poll up to 60s), confirm. Supports verify_change, max_retries (0..=3), settle_ms, retry_delay_ms, require_change.

desktop

list_applications, launch_application, activate_application, open_url, visual_click, double_click, right_click, hover, drag.

screenshot

Unannotated PNG capture of the largest window of a permitted running app after safety preflight.

workflow_memory

list, lookup, record, replay (with optional step_args), clear stored in ~/.apexuse/memory/recipes.json.


Policy, Safety & Environment Variables

Variable

Default

Purpose

APEXUSE_POLICY

./policy.yaml (then ancestor search)

Policy YAML (policy.yaml or policy.autonomous.yaml).

APEXUSE_STATE_DIR

~/.apexuse

Directory for logs/audit.jsonl, memory/recipes.json, and STOP.

APEXUSE_MOCK

0

Set to 1 (or --mock) to run the deterministic in-memory MockDriver.

(Legacy HANDSMCP_* env vars remain supported as fallbacks.)

Built-in safety guardrails enforced regardless of policy:

  • Protected app denial (password, 1password, bitwarden, keychain, wallet, bank, etc.)

  • Native password / secure text field redaction (AXSecureTextField, CurrentIsPassword, AT-SPI PASSWORD_TEXT, bullet-mask checks)

  • Clipboard paste blocking (Cmd+V / Ctrl+V rejected; use computer(action="type"))

  • Emergency kill switch: create ~/.apexuse/STOP to halt all desktop mutations immediately


FAQ

Does ApexUse need an OpenAI, Anthropic, or Google API key? No. apexuse-mcp is a pure local MCP stdio binary with zero built-in LLM calls. Your existing AI agent (Claude Code, Codex, Gemini CLI, Antigravity, etc.) drives it directly over MCP.

How does ApexUse avoid the slow screenshot loop of traditional Computer Use? First, inspect_app reads compact semantic UI elements (e1..eN) without image encoding. Second, visual_inspect lets you set ocr="off" for instant AX-only visual marks. Third, computer(action="run_actions") batches up to 12 deterministic clicks, keystrokes, scrolls, and waits inside a single turn, and computer(action="wait_for") polls locally instead of burning model turns.

Does it hijack my physical mouse cursor while working? On macOS, ApexUse renders an independent translucent AgentCursor overlay window and dispatches targeted events without warping your hardware cursor during normal visual feedback. On Windows, a layered click-through AgentCursor overlay tracks automated actions.

How is ApexUse different from Playwright or browser DevTools MCP servers? Browser MCPs only control a single automated browser tab via CDP. ApexUse controls the real desktop across native OS apps, Electron apps, terminals, dialogs, and user browsers (Chrome, Dia, Safari, Firefox, Edge).


Testing

# Rust unit & integration tests (38 tests)
cargo test --manifest-path rust/Cargo.toml

# End-to-end MCP stdio integration tests (requires python3 mcp + pytest)
.venv/bin/pytest -q

Layout

├── .agents/skills/apexuse/SKILL.md  # Agent skill instructions for ApexUse MCP tools
├── .github/workflows/ci.yml         # 3-OS CI matrix (macOS, Ubuntu Xvfb/AT-SPI, Windows)
├── .github/workflows/release.yml    # Automated macOS/Linux/Windows release binaries
├── agent-configs/README.md          # Client setup snippets (Antigravity, Gemini, Codex, Claude)
├── llms.txt                         # AI search & crawler index summary
├── policy.yaml                      # Default direct policy
├── policy.autonomous.yaml           # Autonomous direct preset
├── pyproject.toml                   # Python MCP E2E test configuration
├── server.json                      # Official MCP Registry manifest
├── rust/
│   ├── Cargo.toml                   # apexuse-mcp crate (0 external dependencies)
│   ├── build.rs                     # Compiles Objective-C macOS bridge on macOS
│   ├── native/macos_bridge.{h,m}    # AXUIElement, SCKit, Vision OCR, AgentCursor, events
│   └── src/
│       ├── driver/                  # OsDriver trait + common.rs + macos.rs + windows.rs + linux.rs
│       ├── gateway.rs               # Snapshots, policy enforcement, verify_change, run_actions
│       ├── json.rs                  # Zero-dependency JSON parser/serializer
│       ├── mcp.rs                   # JSON-RPC 2.0 MCP stdio server & tool schemas
│       ├── memory.rs                # Structural workflow memory
│       ├── policy.rs                # App/field security filters, rate limiter, confirmations
│       ├── som.rs                   # Set-of-Mark AX + OCR fusion
│       └── som_png.rs               # Pure-Rust PNG codec, inflate, pixel digest & SoM renderer
└── tests/test_rust_mcp.py           # End-to-end MCP stdio integration tests

Registry & License

  • MIT License — see LICENSE

  • Official MCP Registry: mcp-name: io.github.Jackpkn/apexuse

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Lets AI agents see and control desktop applications through the accessibility layer, enabling clicking, typing, scrolling, dragging, and window/app management across macOS, Windows, and Linux entirely on the local machine.
      25
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to automate real desktop applications across Windows, Linux, and macOS using incremental screen perception, accessibility trees, OCR, and window management, dramatically reducing token usage compared to screenshot-per-step approaches.
      36 PyPI
      4
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI agents to safely and truthfully control local desktop applications via native accessibility trees, supporting element inspection and automation without proprietary binaries or fake success responses.
      Apache 2.0
    • A
      license
      A
      quality
      B
      maintenance
      Enables AI agents to automate GUI interactions across Windows, Linux, macOS, Android, iOS, and browsers using structured platform APIs first, with vision grounding as a fallback for robust cross-platform computer use.
      17
      MIT