Chatuse

Open-source computer use for macOS, with a visible pointer and no per-app approval layer.

Chatuse lets an AI assistant inspect and operate Mac apps through 21 Model Context Protocol (MCP) tools, or lets you invoke the same operations from a CLI. It combines native accessibility trees, screenshots, on-device OCR, keyboard and mouse input, and window management.

All apps are allowed by default. You grant the normal macOS Accessibility and Screen Recording permissions once during setup; Chatuse does not add app allowlists or ask for consent each time it switches apps. Your MCP host retains its own authorization rules.

Status: early release, built and exercised on an Intel Mac running macOS 15.7.9. The source targets macOS 14+ and builds for the host architecture. Apple Silicon is a supported build target but has not yet been validated on hardware by this project. There is no Linux or Windows backend.

Installation · MCP integration · How it works · Tools · Development · MIT license

Why we made it

This project began when ChatGPT's Locked use setting failed on our Intel development Mac. While investigating, we found that the Intel app package we examined did not contain the native computer-use helper needed by that setup. We wanted an implementation we could inspect, build, change, and use with our own assistant.

The goals became straightforward:

• Make native Mac app control available through an open, local tool interface.

• Allow the assistant to work across apps without a separate per-app approval layer.

• Make its intended pointer movements and clicks visible on the screen.

• Keep the implementation small enough to understand, test, and extend.

• Publish the source under MIT so others can adapt it to their own workflows.

Chatuse is an independent implementation using public macOS APIs. It is not affiliated with or endorsed by OpenAI, and contains no OpenAI native helper code or binaries. It does not modify ChatGPT or enable its existing Locked use toggle. It currently operates in your unlocked desktop session; an isolated desktop and session unlocking are outside this release's capabilities.

Related MCP server: macbot-mcp

What it can do

• Read app accessibility trees and act on observed buttons, fields, and other controls.

• Capture an app window or display, optionally recognize text with Apple's Vision framework, and map image coordinates back to screen points.

• Type Unicode without replacing the clipboard; send shortcuts, clicks, scrolling, and timed drags.

• Focus, move, resize, minimize, restore, raise, and close windows.

• Discover running apps and monitors, launch installed apps, open HTTP(S) links, and explicitly read or write clipboard text.

• Show a blue pointer with a label and click rings, then fade it away when idle.

• Pause input immediately through an emergency-stop tool or local command.

Chatuse supplies the tools. It does not include an AI model, call a model API itself, require an API key, or run a network server. Your MCP client provides the assistant and decides which observations to send to its model.

Installation

Requirements

If you need Apple's development tools, run xcode-select --install and complete its installer before building.

Clone and build

git clone https://github.com/fazakis/chatuse.git "$HOME/chatuse"
cd "$HOME/chatuse"
npm ci
npm run build
./chatuse setup

The build creates runtime/Chatuse.app and runtime/chatuse-pointer. The app bundle contains both the native helper and its setup window. It is ad-hoc signed for use on the Mac where you build it; no prebuilt or notarized application is distributed here.

Grant macOS permissions

In the Chatuse setup window, open System Settings → Privacy & Security and enable Accessibility and Screen Recording for Chatuse. If necessary, use the settings pane's + button and select Chatuse.app inside this checkout's runtime directory. macOS can attribute access to the app launching the helper; follow the app name the system actually presents.

Restart the MCP connection or quit and reopen its host if macOS requests it. Then check:

./chatuse status

For the full toolset, expect accessibility and screenRecording to be true, and locked, secureInput, and stopped to be false. Some discovery operations work without both permissions. ./chatuse permissions invokes the normal system permission flow; it does not grant permissions itself.

MCP integration

Codex

After building, register the server:

sh scripts/install-codex.sh

The script uses codex mcp add with the checkout's absolute path and leaves an existing chatuse registration intact. It can use the Codex executable bundled in ChatGPT or one on your PATH. Set CHATUSE_CODEX_BIN to an explicit executable path if needed.

Restart the MCP connection or host so it discovers the tools. Try a first request:

Use Chatuse to check its permissions, inspect Finder, and show me its current window.

Other MCP clients

Add a stdio server using the absolute path to your checkout's launcher:

{
  "mcpServers": {
    "chatuse": {
      "command": "/absolute/path/to/chatuse/chatuse",
      "args": ["mcp"]
    }
  }
}

Do not use ~ in a command field unless your client explicitly expands it. The build saves the selected Node executable as the runtime/node symlink so GUI hosts do not have to inherit your shell's PATH.

The checked-in .mcp.json is another option: it uses /bin/sh to launch $HOME/chatuse/chatuse, or ${CHATUSE_ROOT}/chatuse when you set CHATUSE_ROOT in the server's environment. It works independently of the host's working directory. For example, an MCP client can add "env": {"CHATUSE_ROOT": "/absolute/path/to/chatuse"} to that server entry.

A Codex plugin manifest and computer-use skill are included for local packaging. That package points to the built checkout through .mcp.json; it does not download dependencies or build the native helper. Direct MCP registration is the tested setup path. Use one connection method to avoid duplicate tool registrations.

How it works

flowchart TD
    A[Assistant in an MCP host] -->|stdio MCP| B[Node.js server]
    C[CLI] --> B
    B -->|Serialized JSON requests| D[Swift native helper]
    B -->|Visual feedback| E[Swift pointer overlay]
    D --> F[Accessibility: inspect and act]
    D --> G[ScreenCaptureKit and Vision: capture and OCR]
    D --> H[AppKit and Core Graphics: apps, windows, and input]
    E --> I[Click-through panel on the desktop]

The Node.js layer validates tool arguments, manages a persistent native process, coordinates pointer feedback, and returns text and images through MCP. The Swift helper talks to macOS through public frameworks. Communication between these processes is local stdin/stdout; there is no listening port.

A typical interaction follows this loop:

1. Check and discover: call chatuse_status, then find the app with chatuse_list_apps.

2. Observe: call chatuse_observe with the app's bundle ID or PID.

3. Act: prefer an accessibility action using the returned snapshotId and elementId. For visual targets, use pixels from the returned screenshot and supply its screenshotId.

4. Verify: observe again and check the resulting state before continuing.

For example, these are arguments to chatuse_observe, an accessibility chatuse_click, and a coordinate chatuse_click, respectively. Replace reference IDs and coordinates with values from actual observations:

{"app":"com.apple.finder","screenshot":true,"ocr":false}

{"snapshotId":"ID_FROM_INSPECT","elementId":"ELEMENT_FROM_INSPECT"}

{"screenshotId":"ID_FROM_SCREENSHOT","x":420,"y":160}

Snapshot references belong to one native-helper session, expire after 120 seconds, and live in bounded caches of eight accessibility snapshots and eight screenshots. Window screenshots are checked for changed geometry before coordinate input. Global display screenshots have no per-window geometry guarantee. Screen points can have negative coordinates on multi-monitor setups; screenshot scaling and origins are included in the mapping.

Accessibility actions can operate in the background when an app supports them. Mouse and keyboard actions use the foreground desktop. Text input checks focus between chunks and dragging checks it between steps. Accessibility data, screenshots, and OCR are gathered at different instants, so the assistant must verify results. If a request times out or disconnects, Chatuse does not automatically replay the action.

Tools

Every MCP tool name begins with chatuse_. The table uses the shorter CLI names.

Argument schemas and descriptions live in server/tools.mjs. list_apps lists running apps; launching an app requires a known installed bundle ID or absolute .app path.

CLI

Run commands from the checkout, or use the launcher's absolute path:

./chatuse help
./chatuse status
./chatuse list_apps
./chatuse inspect '{"app":"com.apple.finder","maxDepth":10}'
./chatuse screenshot '{"app":"com.apple.finder","ocr":true}'
./chatuse press_key '{"app":"com.apple.finder","key":"n","modifiers":["cmd"]}'
./chatuse pointer-demo
./chatuse pointer off
./chatuse pointer on
./chatuse stop
./chatuse resume
./chatuse mcp

Each ordinary CLI invocation starts a fresh helper, so use an MCP session for multi-step actions that depend on snapshot IDs. CLI screenshots are written to artifacts/ with user-only file permissions. MCP screenshots are returned in memory without being saved by Chatuse.

Visible pointer

The blue, labeled overlay shows where Chatuse intends to act. It animates toward observed targets before clicks, shows expanding click rings, follows drag gestures, and indicates scrolling when target coordinates are supplied. Accessibility clicks get feedback when the current client has observed nonzero bounds for their element.

The overlay ignores mouse events and cannot take keyboard focus. It fades after about three seconds of inactivity and hides when the Mac is locked or emergency stop is active. It is suppressed during Chatuse screenshots so the assistant does not mistake the pointer for app content. The animation shows the attempted action; it cannot prove the app accepted it.

./chatuse pointer-demo demonstrates the overlay without moving or clicking the real mouse. pointer off disables only the visual feedback. A targeted pointer action normally adds about 320 ms of animation before input.

Privacy and control

Chatuse runs with your user account's desktop access. All apps are allowed by its policy, but macOS permissions, app accessibility support, and the MCP host's authorization rules still determine which actions work.

• System permissions: Chatuse uses the normal Accessibility and Screen Recording flows. It does not modify TCC databases or unlock a locked session. It refuses new input while macOS Secure Input is active and redacts secure accessibility field values.

• Observations: screenshots, OCR, accessibility data, and clipboard reads can contain private material visible to your account. Your MCP host may send tool results to its model and retain them in conversation history. Chatuse itself makes no model requests and sends no telemetry.

• Audit log: runtime/audit.jsonl records operation names, timestamps, duration, and result codes. It omits arguments, typed text, app contents, clipboard values, and images. It rotates at roughly 2 MB with one backup.

• Serialization: native operations are serialized per helper, and an OS file lock prevents simultaneous input from multiple Chatuse helper processes using the same checkout.

• Emergency stop: ./chatuse stop, the local setup window, or chatuse_emergency_stop creates the checkout's STOP file. This pauses input, launches, URL opening, and clipboard writes; read-only inspection remains available when the session permits. In-progress text and drag operations stop at their next check. Resume from the local setup window or ./chatuse resume; there is no MCP resume tool.

• App content: the bundled skill tells the assistant to treat screen and app text as task data, never as instructions that override the user.

Stopping cannot undo an action that already completed. Normal cancellation requests release held mouse input when the helper can handle termination; force-killing a process or a blocked operating-system API cannot guarantee that cleanup.

Development

npm ci
npm run build
npm test
npm run test:native

npm test includes a real MCP handshake with the built native helper, so run the build first on macOS. The suite covers native transport, errors, timeouts, cancellation, no automatic action replay, audit-log privacy, stop responsiveness, input validation, and pointer coordination. Swift tests cover coordinate mapping, reference expiry, and Unicode boundaries.

For desktop integration tests, grant both macOS permissions, unlock the Mac, and leave the keyboard and mouse idle while the harness uses its own test window:

npm run test:e2e
npm run test:pointer

The desktop harness checks accessibility actions, Unicode typing, screenshots/OCR, coordinate clicks, stale screenshot rejection, window operations, and emergency stop. It writes artifacts/e2e-report.json and exits 2 with a blocked report when desktop prerequisites are absent. The pointer harness checks visibility, input transparency, focus preservation, screenshot suppression, and idle hiding. Generated reports, screenshots, binaries, logs, and runtime state are excluded from Git.

At initial publication, validation on the Intel development Mac passed 18 Node tests, 6 Swift tests, 10 live desktop checks, and 5 pointer checks. Those results describe that environment, not a certification of every Mac or third-party app.

See CONTRIBUTING.md for the contribution workflow.

Updating and troubleshooting

Update a source installation with git pull --ff-only, npm ci, and npm run build, then restart the MCP connection or host. Full builds replace the ad-hoc-signed helper, which may require granting macOS permissions again. Set CHATUSE_SIGN_IDENTITY to your own signing identity if you have one; signing and notarization for distribution remain your responsibility.

For a change confined to the pointer, run sh scripts/build-pointer.sh. This rebuilds only runtime/chatuse-pointer and leaves the authorized Chatuse.app bundle untouched. Restart the MCP host after JavaScript changes so existing server processes load them.

press_key uses US physical key positions; use type_text for arbitrary Unicode. OCR can be imperfect. Multi-window apps are best captured with an explicit windowId; accessibility window operations use a separate zero-based AX window index.

To disconnect Codex, run codex mcp remove chatuse using the same Codex executable used to register it. For other clients, remove the server entry. Revoke Chatuse's system permissions if no longer needed. The checkout and generated runtime remain until you remove them.

Contributing and license

Bug reports, reproducible app-compatibility examples, Apple Silicon validation, and focused pull requests are welcome through GitHub issues and pull requests. Please remove private screen contents and identifiers from any reports you share.

Chatuse's original source is licensed under the MIT License, copyright © 2026 Chatuse contributors. You may use, modify, redistribute, and use it commercially under that license. Third-party dependencies retain their own licenses.