Which integrations are available for this server?

Provides desktop automation on macOS via Accessibility and Screen Capture, enabling AI agents to control allowed applications with policy enforcement, audit logging, crash recovery, and task journaling.

How do I use agent-computer-use?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@agent-computer-use Open Calculator and calculate 5 + 7" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

agent-computer-use

by waniyaro

Overview Schema Source Related Servers Discussions Admin

TypeScript

Local

🖥️ /agent-computer-use

Experimental Lightweight Safety, Policy Enforcement, and Crash Recovery Middleware for Desktop AI Agents on macOS

License: MIT Platform: macOS TypeScript MCP Version CI Status

English • Русский

agent-computer-use is an open-source, experimental Model Context Protocol (MCP) proxy server built on top of Cua Driver.

While raw computer-use drivers provide low-level capabilities to click and type, agent-computer-use provides a dedicated security and reliability proxy layer: strict per-application bundle ID access control, fail-closed enforcement, instant kill-switches, sanitized audit logging with execution timing, process cache invalidation, crash recovery, and durable task journaling for autonomous agents.

🏛️ Architecture

graph TD
    classDef client fill:#1e293b,stroke:#3b82f6,stroke-width:2px,color:#f8fafc;
    classDef proxy fill:#0f172a,stroke:#6366f1,stroke-width:2px,color:#f8fafc;
    classDef module fill:#1e1b4b,stroke:#818cf8,stroke-width:1px,color:#e0e7ff;
    classDef driver fill:#064e3b,stroke:#10b981,stroke-width:2px,color:#ecfdf5;
    classDef app fill:#450a0a,stroke:#ef4444,stroke-width:2px,color:#fef2f2;

    Client["<b>AI Agent / MCP Client</b><br/>Antigravity IDE · Claude Code · Codex · Cursor"]:::client

    subgraph ACU ["agent-computer-use (Proxy Server)"]
        direction TB
        subgraph Safety ["Security & Observability Layer"]
            Enforcer["<b>PolicyEnforcer</b><br/>• Fail-Closed Bundle Allowlist<br/>• Strict Bundle ID Verification<br/>• PID Cache Invalidation<br/>• Live STOP File Kill-Switch"]:::module
            Audit["<b>AuditLogger</b><br/>• Masked Typed Text<br/>• Binary/Screenshot Stripping<br/>• Duration ms Tracking<br/>• JSONL 10MB Auto-Rotation"]:::module
            Journal["<b>JournalManager</b><br/>• Session Task Journaling<br/>• Context Loss Recovery"]:::module
        end
        Recovery["<b>AppRecoveryManager</b><br/>• ensure_app_running self-healing<br/>• autoRelaunch detection"]:::module
    end

    Driver["<b>Cua Driver (stdio daemon)</b><br/>macOS Accessibility (AXUIElement) · Screen Capture"]:::driver
    Apps["<b>Target Desktop Applications</b><br/>Calculator · Browsers · Native Apps"]:::app

    Client <-->|"stdio (JSON-RPC)"| ACU
    Safety --> Recovery
    ACU <-->|"stdio (JSON-RPC)"| Driver
    Driver <-->|"AXEvents (Background Delivery Mode)"| Apps

Related MCP server: Daimon

✨ Key Features

Capability	What It Does	Why It Matters
🔒 Fail-Closed Security	Blocks all GUI interactions unless an app's macOS Bundle ID is explicitly in `allowedApps`.	Prevents rogue AI agents from wandering into unauthorized desktop windows.
⛔ Protected App Denylist	Hard-blocks password managers (1Password, Bitwarden, Keychain), System Settings, and Terminals by bundle ID.	Denylist takes priority. Process name spoofing is strictly prevented.
🛑 Instant Kill-Switch	Creates `~/.config/agent-computer-use/STOP` to immediately freeze all actions.	Halts runaway agents on the fly without needing to kill IDE processes.
🤫 Private Audit Log	Writes JSONL events with duration metrics, masked keystrokes (`logTypedText: false`), and stripped screenshots.	Full observability and security audit trails without leaking credentials.
🔄 Self-Healing Recovery	Custom `ensure_app_running` tool detects closed or crashed windows and restarts them, invalidating stale PID caches.	Agents recover automatically without throwing raw OS errors or getting stuck.
📝 Durable Task Journal	Persistent JSONL journal (`task_journal_*`) per session with pagination/limit support.	Survives agent memory loss, context compaction, and IDE restarts.
⚡ Minimal Tool Profile	Filters Cua Driver's 58 tools into 16 focused, reliable primitives.	Keeps agent prompt context small, prevents tool confusion, and improves LLM reasoning.

🚀 Quick Start

1. Prerequisites

OS: macOS 14+ (Sonoma, Sequoia, or newer; Apple Silicon & Intel) as required by Cua Driver.
Node.js: v20+ (v22 LTS recommended).
Cua Driver: Installed following the official Cua installation guide. (Always inspect installation scripts before running locally).

2. Installation

git clone https://github.com/waniyaro/agent-computer-use.git
cd agent-computer-use
npm install
npm run build

3. Run System Diagnostics (`acu doctor`)

node dist/bin/acu.js doctor

The doctor verifies macOS version, Cua Driver binary, Accessibility & Screen Recording permissions, policy schema validity, and client integrations:

======================================================
           agent-computer-use Doctor Report           
======================================================
--- 1. macOS Environment ---
  [OK]   Operating System: macOS 26.6.2 (arm64)
--- 2. Cua Driver ---
  [OK]   Binary executable: ~/.local/bin/cua-driver (cua-driver 0.34.0)
--- 3. macOS Permissions ---
  [OK]   TCC Grants: Accessibility: granted, Screen Recording: granted
--- 4. Policy Configuration ---
  [OK]   policy.json validation: Valid (toolProfile='minimal', allowedApps=1)
--- 5. Kill-Switch Status ---
  [OK]   Emergency STOP file: Inactive (normal operation)
--- 6. Client Integrations ---
  [OK]   Antigravity IDE: Configured in ~/.gemini/config/mcp_config.json
  [OK]   Claude Code: Configured in ~/.claude.json
  [OK]   Codex: Not installed (optional)
------------------------------------------------------
Overall Status: [OK]
======================================================

4. Connect to Your MCP Client (`acu install`)

Install into your AI IDE with safe dry-run preview and automatic timestamped backups:

# Preview configuration (safe dry-run)
node dist/bin/acu.js install --client antigravity --dry-run

# Write configuration (creates .bak copy and preserves all other MCP servers)
node dist/bin/acu.js install --client antigravity --write

Supported clients:

antigravity (Google Antigravity IDE)
claude-code (Anthropic Claude Code CLI)
codex (Codex CLI)

🛡️ Policy Configuration (`policy.json`)

Location: ~/.config/agent-computer-use/policy.json

{
  "allowedApps": [
    "com.apple.calculator"
  ],
  "deniedApps": [
    "com.1password.1password",
    "com.agilebits.onepassword7",
    "com.bitwarden.desktop",
    "com.apple.keychainaccess",
    "com.apple.systempreferences",
    "com.apple.Terminal",
    "com.googlecode.iterm2"
  ],
  "maxActionsPerSession": 200,
  "toolProfile": "minimal",
  "allowForeground": false,
  "logTypedText": false,
  "autoRelaunch": false,
  "allowAnyApp": false
}

Configuration Reference

Option	Type	Default	Description
`allowedApps`	`string[]`	`[]`	Whitelist of application bundle IDs (e.g., `"com.apple.calculator"`). When empty, all action tools are blocked (Fail-Closed). Wildcards are not supported to prevent bypasses.
`deniedApps`	`string[]`	`[...]`	Blacklist of sensitive apps by bundle ID. Always evaluated before `allowedApps`.
`maxActionsPerSession`	`number`	`200`	Safety cap preventing infinite loops or hallucinated repetitive clicks.
`toolProfile`	`"minimal" \| "full"`	`"minimal"`	`"minimal"` exposes 16 essential tools; `"full"` exposes all 58 backend tools.
`allowForeground`	`boolean`	`false`	When `false`, blocks `delivery_mode: "foreground"` to prevent focus stealing.
`logTypedText`	`boolean`	`false`	When `false`, masks input text in audit logs to protect credentials.
`autoRelaunch`	`boolean`	`false`	Automatically restarts the target application when a crash is detected.
`allowAnyApp`	`boolean`	`false`	Disables allowlist checks (Not recommended). Denylist is still enforced.

IMPORTANT

GUI Security Boundary vs. Agent Host Environment:
The deniedApps list and Policy Enforcer strictly govern GUI automation actions executed through this MCP proxy. They prevent the agent from viewing, clicking, or typing into protected macOS GUI windows (such as Passwords, Keychain, System Settings, Script Editor, Terminal.app, or iTerm2).
This proxy does NOT sandbox the agent's host environment or other IDE tools. If an agent running in an IDE (e.g. Antigravity, Claude Code, Cursor) has access to shell execution tools (run_command, bash, terminal tools), it executes commands directly on the operating system outside this proxy. Blocking com.apple.Terminal in deniedApps prevents the agent from manipulating the Terminal window via GUI clicks/keystrokes, but does not restrict background shell commands run by the agent through other MCP servers or IDE capabilities.

Default Denied Applications:

com.1password.1password (1Password)
com.agilebits.onepassword7 (1Password 7)
com.bitwarden.desktop (Bitwarden)
com.apple.keychainaccess (Keychain Access)
com.apple.Passwords (macOS Passwords app)
com.apple.systempreferences (macOS System Settings)
com.apple.Terminal (Terminal.app)
com.googlecode.iterm2 (iTerm2)
com.apple.ScriptEditor2 (Script Editor)

Emergency Kill-Switch

To instantly freeze all actions without restarting your IDE:

touch ~/.config/agent-computer-use/STOP

To resume:

rm ~/.config/agent-computer-use/STOP

🛠️ Custom Proxy Tools

In addition to proxying Cua Driver primitives (click, type_text, scroll, get_window_state), agent-computer-use introduces 4 high-level reliability tools:

`ensure_app_running`

Verifies whether an application process and window exist. If closed or crashed, launches it in the background, waits for window initialization, and invalidates stale process caches.

{
  "bundle_id": "com.apple.calculator",
  "timeout_ms": 5000
}

`task_journal_append`

Appends a milestone, observation, or status update to the session's JSONL journal (~/.config/agent-computer-use/journal/<task_id>.jsonl).

{
  "note": "Calculated 17 * 23 = 391",
  "status": "completed"
}

`task_journal_read`

Reads past checkpoints (supports optional limit or task_id; defaults to current session) so an agent can resume trajectories after compaction or restart.

{
  "limit": 20
}

`task_journal_list`

Lists all historical and active task journals with summary statistics.

🧪 Testing

The test suite runs with Vitest and validates proxy resilience, policy enforcement, audit sanitization, task journaling, and CLI commands:

npm test

 ✓ tests/audit.test.ts (5 tests)
 ✓ tests/cli.test.ts (9 tests)
 ✓ tests/journal.test.ts (5 tests)
 ✓ tests/policy.test.ts (10 tests)
 ✓ tests/proxy.test.ts (4 tests)
 ✓ tests/recovery.test.ts (4 tests)

 Test Files  6 passed (6)
      Tests  37 passed (37)

🤝 Contributing

Contributions are warmly welcome! Please review CONTRIBUTING.md and SECURITY.md before opening a pull request.

📄 License

This project is licensed under the MIT License.
Underlying desktop automation powered by Cua Driver (MIT License).

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

Need Help?

Related Servers

Related MCP Connectors

IdentArk Gateway
io.identark
Zero-secret MCP gateway for AI agents: risk-scored, audited calls with human-in-the-loop approval.
Security Agent Orchestration
29
Cotlet C
dev.cotlet
Hosted MCP server and REST API that lets AI agents use apps with no API, signed in as you.
Browser Automation App Automation Agent Orchestration
4661
agentlock
io.github.Olum289
Security gateway for AI agents: policy, approval, and audited execution, no secrets shared.
Security Agent Orchestration
2
ActionDock A
app.actiondock
Supervised API-write gateway for AI agents with policy, human approval and execution receipts.
Agent Orchestration Security App Automation
81

Related MCP Servers

macinput
OS Automation App Automation
Sigma711
A
license
Not graded
quality
D
maintenance
An MCP server for macOS that enables AI agents to control the desktop GUI through keyboard input, mouse actions, and screen captures. It provides stable low-level primitives for UI automation and agent-driven desktop workflows.
Updated 7 months ago (2026-03-19 09:29 UTC)
MIT
Daimon
OS Automation Autonomous Agents
ArboRithm
F
license
Not graded
quality
B
maintenance
A local daemon for macOS that gives any MCP-capable AI client eyes, hands, and a face — screen capture, accessibility tree, mouse/keyboard actions, and an overlay — with a built-in security ceiling.
Updated 8 days ago (2026-10-15 15:10 UTC)
-
macos-sys-assist
OS Automation File Systems
kbdiamondes
A
license
Not graded
quality
D
maintenance
A secure, constraint-based macOS OS-level automation MCP server for AI assistants.
Updated 4 months ago (2026-06-25 08:23 UTC)
MIT
Hands
OS Automation Autonomous Agents
yuvitbatra
A
license
A
quality
D
maintenance
A macOS computer-use MCP server that grants AI agents mouse, keyboard, and screen control with a robust security model including permission profiles, app deny-lists, and audit logging.
Updated 4 months ago (2026-07-06 08:39 UTC)
22
MIT

🖥️ /agent-computer-use

🏛️ Architecture

✨ Key Features

🚀 Quick Start

1. Prerequisites

2. Installation

3. Run System Diagnostics (acu doctor)

4. Connect to Your MCP Client (acu install)

🛡️ Policy Configuration (policy.json)

Configuration Reference

Default Denied Applications:

Emergency Kill-Switch

🛠️ Custom Proxy Tools

ensure_app_running

task_journal_append

task_journal_read

task_journal_list