ApexUse
Allows agents to inspect and automate Electron-based desktop applications through the OS accessibility tree and visual grounding, treating them like any other native app.
Enables agents to control the Firefox browser window on the desktop using accessibility refs, visual marks, and synthesized input, alongside other user browsers.
Provides desktop control on Linux under X11 and Wayland, using AT-SPI2 accessibility trees, libX11/libXtst input, tesseract OCR, and Wayland screenshot/input helpers to inspect and automate native Linux applications.
Provides full desktop control on macOS 14+ via native Accessibility (AXUIElement) trees, ScreenCaptureKit screenshots, Apple Vision OCR, and per-PID CGEvent/NSEvent input, letting agents inspect, click, type, scroll, and verify changes in native macOS apps.
Enables agents to drive the Safari browser on the desktop through the same accessibility, visual Set-of-Mark, and input tooling used for native apps, rather than a browser-specific automation protocol.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
ApexUse — Computer Use MCP Server for AI Agents (macOS, Windows & Linux)
ApexUse (apexuse-mcp) is an open-source, local-first Computer Use MCP server written in pure Rust and native OS bridges. It gives AI coding agents — including Claude Code, OpenAI Codex, Gemini CLI, and Antigravity — full desktop and browser control over macOS, Windows, and Linux using native Accessibility trees, Set-of-Mark ([m1]..[mN]) visual grounding, local OCR, a non-hijacking AgentCursor overlay, batched run_actions, and local workflow memory.
Zero external LLM keys. Zero external Rust crate dependencies. Zero cloud relay.
Why ApexUse vs. Alternative Computer-Use Tools?
Capability | ApexUse ( | Screenshot-Only Computer Use | Browser-Only MCPs (Playwright / DevTools) |
App Coverage | Full OS: native apps, Electron, terminals & any browser | Whole screen | Browser tabs only |
UI Grounding | Hybrid: semantic Accessibility ( | Pixel coordinates only | DOM only |
Multi-Step Speed |
| 1 LLM round-trip per click | Step-by-step tool calls |
Scroll & Verification | Target-scoped | Blind scroll + manual re-screenshot | Page scroll only |
Visual Pointer | Floating transparent | Teleports real user cursor | None / page focus |
Workflow Memory | Records & replays structural macros via | Re-plans every run | Scripts must be coded |
Security Controls | Native password/secure-field blocking, protected-app list, | Sends full raw screen pixels | Web origin scope |
Related MCP server: oswright
Platform Support
Platform | Backend Driver | Capabilities & Verification |
macOS 14+ |
|
|
Windows 10 / 11 |
| Pure-Rust Win32 FFI ( |
Linux (X11 & Wayland) |
| Runtime |
Quick Start
1. Install or build from source
Download prebuilt binaries for macOS (arm64), Linux (x86_64), or Windows (x86_64) from GitHub Releases, or install directly from crates.io:
cargo install apexuse-mcpOr clone and build directly (requires only the Rust toolchain, plus Xcode Command Line Tools on macOS):
git clone https://github.com/Jackpkn/apexuse.git
cd apexuse
cargo build --release --manifest-path rust/Cargo.tomlOutput binary: rust/target/release/apexuse-mcp (or apexuse-mcp.exe on Windows).
2. Check OS readiness & permissions
./rust/target/release/apexuse-mcp doctormacOS: Enable Accessibility and Screen Recording for the terminal or IDE launching
apexuse-mcpin System Settings → Privacy & Security.Windows: Uses built-in Win32 / UI Automation / WinRT APIs with no extra dependencies.
Linux: Runs on X11 via
libX11/libXtst(dlopen). For full semantic accessibility, OCR, and Wayland support install distro packages as needed (e.g.,python3-gi,gir1.2-atspi-2.0,tesseract-ocr,xdotool, orgrim/wtype/ydotoolon Wayland).
3. Connect to your AI coding agent
Generate a ready-to-paste snippet for your client:
./rust/target/release/apexuse-mcp install antigravity
# Supported: antigravity | gemini | claude | codexClaude Code:
claude mcp add apexuse --env APEXUSE_POLICY=/abs/path/to/apexuse/policy.autonomous.yaml -- /abs/path/to/apexuse/rust/target/release/apexuse-mcpAntigravity / Gemini CLI (
~/.gemini/antigravity/mcp_config.jsonor~/.gemini/settings.json):{ "mcpServers": { "apexuse": { "command": "/abs/path/to/apexuse/rust/target/release/apexuse-mcp", "args": [], "env": { "APEXUSE_POLICY": "/abs/path/to/apexuse/policy.autonomous.yaml" } } } }OpenAI Codex (
~/.codex/config.toml):[mcp_servers.apexuse] command = "/abs/path/to/apexuse/rust/target/release/apexuse-mcp" args = [] startup_timeout_sec = 60 tool_timeout_sec = 180 [mcp_servers.apexuse.env] APEXUSE_POLICY = "/abs/path/to/apexuse/policy.autonomous.yaml"
See agent-configs/README.md and the bundled skill at .agents/skills/apexuse/SKILL.md.
MCP Tools Reference (6 Tools)
Tool | Actions & Key Parameters |
| Read-only Accessibility snapshot for |
| Captures an app window or full |
|
|
|
|
| Unannotated PNG capture of the largest window of a permitted running app after safety preflight. |
|
|
Policy, Safety & Environment Variables
Variable | Default | Purpose |
|
| Policy YAML ( |
|
| Directory for |
|
| Set to |
(Legacy HANDSMCP_* env vars remain supported as fallbacks.)
Built-in safety guardrails enforced regardless of policy:
Protected app denial (
password,1password,bitwarden,keychain,wallet,bank, etc.)Native password / secure text field redaction (
AXSecureTextField,CurrentIsPassword,AT-SPI PASSWORD_TEXT, bullet-mask checks)Clipboard paste blocking (
Cmd+V/Ctrl+Vrejected; usecomputer(action="type"))Emergency kill switch: create
~/.apexuse/STOPto halt all desktop mutations immediately
FAQ
Does ApexUse need an OpenAI, Anthropic, or Google API key?
No. apexuse-mcp is a pure local MCP stdio binary with zero built-in LLM calls. Your existing AI agent (Claude Code, Codex, Gemini CLI, Antigravity, etc.) drives it directly over MCP.
How does ApexUse avoid the slow screenshot loop of traditional Computer Use?
First, inspect_app reads compact semantic UI elements (e1..eN) without image encoding. Second, visual_inspect lets you set ocr="off" for instant AX-only visual marks. Third, computer(action="run_actions") batches up to 12 deterministic clicks, keystrokes, scrolls, and waits inside a single turn, and computer(action="wait_for") polls locally instead of burning model turns.
Does it hijack my physical mouse cursor while working?
On macOS, ApexUse renders an independent translucent AgentCursor overlay window and dispatches targeted events without warping your hardware cursor during normal visual feedback. On Windows, a layered click-through AgentCursor overlay tracks automated actions.
How is ApexUse different from Playwright or browser DevTools MCP servers? Browser MCPs only control a single automated browser tab via CDP. ApexUse controls the real desktop across native OS apps, Electron apps, terminals, dialogs, and user browsers (Chrome, Dia, Safari, Firefox, Edge).
Testing
# Rust unit & integration tests (38 tests)
cargo test --manifest-path rust/Cargo.toml
# End-to-end MCP stdio integration tests (requires python3 mcp + pytest)
.venv/bin/pytest -qLayout
├── .agents/skills/apexuse/SKILL.md # Agent skill instructions for ApexUse MCP tools
├── .github/workflows/ci.yml # 3-OS CI matrix (macOS, Ubuntu Xvfb/AT-SPI, Windows)
├── .github/workflows/release.yml # Automated macOS/Linux/Windows release binaries
├── agent-configs/README.md # Client setup snippets (Antigravity, Gemini, Codex, Claude)
├── llms.txt # AI search & crawler index summary
├── policy.yaml # Default direct policy
├── policy.autonomous.yaml # Autonomous direct preset
├── pyproject.toml # Python MCP E2E test configuration
├── server.json # Official MCP Registry manifest
├── rust/
│ ├── Cargo.toml # apexuse-mcp crate (0 external dependencies)
│ ├── build.rs # Compiles Objective-C macOS bridge on macOS
│ ├── native/macos_bridge.{h,m} # AXUIElement, SCKit, Vision OCR, AgentCursor, events
│ └── src/
│ ├── driver/ # OsDriver trait + common.rs + macos.rs + windows.rs + linux.rs
│ ├── gateway.rs # Snapshots, policy enforcement, verify_change, run_actions
│ ├── json.rs # Zero-dependency JSON parser/serializer
│ ├── mcp.rs # JSON-RPC 2.0 MCP stdio server & tool schemas
│ ├── memory.rs # Structural workflow memory
│ ├── policy.rs # App/field security filters, rate limiter, confirmations
│ ├── som.rs # Set-of-Mark AX + OCR fusion
│ └── som_png.rs # Pure-Rust PNG codec, inflate, pixel digest & SoM renderer
└── tests/test_rust_mcp.py # End-to-end MCP stdio integration testsRegistry & License
MIT License — see LICENSE
Official MCP Registry: mcp-name: io.github.Jackpkn/apexuse
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
Connect ChatGPT, Claude, Cursor or VS Code to your Windows PC: files, terminal, browser, apps.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceLets AI agents see and control desktop applications through the accessibility layer, enabling clicking, typing, scrolling, dragging, and window/app management across macOS, Windows, and Linux entirely on the local machine.25MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to automate real desktop applications across Windows, Linux, and macOS using incremental screen perception, accessibility trees, OCR, and window management, dramatically reducing token usage compared to screenshot-per-step approaches.36 PyPI4MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to safely and truthfully control local desktop applications via native accessibility trees, supporting element inspection and automation without proprietary binaries or fake success responses.Apache 2.0
- AlicenseAqualityBmaintenanceEnables AI agents to automate GUI interactions across Windows, Linux, macOS, Android, iOS, and browsers using structured platform APIs first, with vision grounding as a fallback for robust cross-platform computer use.17MIT