Skip to main content
Glama

Nova

English · 简体中文

Let any MCP agent use your real Mac or Windows apps. Nova is a single Rust binary, with no Python runtime, that gives Claude Desktop, Cursor, Codex, Claude Code and other MCP clients screenshots, native OCR, keyboard and mouse input, clipboard access, window/app control and Accessibility/UIA actions on macOS and Windows.

Releases · Part of Bodhi · Bodhi desktop app · MIT

  • Your real desktop, not a VM: macOS 14+ (Apple Silicon and Intel) and Windows x64/ARM64.

  • Reads before it clicks: Accessibility/UIA controls, Apple Vision or Windows OCR, and per-window screenshots with zoom. Clicks are given in the screenshot's pixel space and mapped back to the screen.

  • Types any text: full Unicode keyboard input, including Chinese, Japanese, Korean and emoji.

  • Accessibility-first control (v0.3.0): AX-first ax_read / ax_activate that reject stale snapshots, a 64-step batch_actions limit with structured failure reports, the Nova.app permission owner (development preview) and an optional Chrome DevTools sidecar. Latest release: v0.3.0 — Releases.

Static image · Reproduce and inspect MCP evidence

This recording shows Nova's browser tool path (fixture page and real MCP navigation/click calls). It was captured on Linux, so it does not show native macOS/Windows desktop control or model reasoning. The reproduction notes explain the --npx adapter and isolated browser.

Install

Platform

Published v0.3.0

macOS 14+ (Homebrew)

brew install bigduu/tap/nova

macOS 14+ (manual)

nova-v0.3.0-universal-apple-darwin.tar.gz from Releases

Windows x86_64

nova-v0.3.0-x86_64-pc-windows-msvc.zip

Windows ARM64

nova-v0.3.0-aarch64-pc-windows-msvc.zip

Linux

Desktop control is not supported (headless protocol checks only)

Homebrew (macOS):

brew tap bigduu/tap
brew install bigduu/tap/nova
nova --version

The formula installs the CLI from the release archive. It does not configure an MCP client or grant macOS permissions. Run which nova to find the path to use below: usually /opt/homebrew/bin/nova on Apple Silicon and /usr/local/bin/nova on Intel Macs.

npm (macOS and Windows): Homebrew stays the recommended macOS install. @bigduu/nova (package version 0.3.1) is a Node launcher that downloads the matching v0.3.1 GitHub Release binary: universal macOS, or Windows x64 / ARM64. It does not install a Linux desktop binary. The MCP Registry name is io.github.bigduu/nova.

npx @bigduu/nova --version
npm i -g @bigduu/nova

An MCP client that cannot answer an install prompt should call npx with -y:

{
  "mcpServers": {
    "nova": { "command": "npx", "args": ["-y", "@bigduu/nova"] }
  }
}

The launcher checks the release archive's .sha256 before writing nova or nova.exe. The macOS binary is ad-hoc signed, not notarized. Windows binaries are not Authenticode-signed, so SmartScreen may warn on first run.

Manual download: download the matching .sha256 file from the same release and verify the archive before extracting it. On macOS:

tar -xzf nova-v*-universal-apple-darwin.tar.gz
xattr -dr com.apple.quarantine ./nova  # only if Gatekeeper blocks the download
sudo install -m 0755 nova /usr/local/bin/nova

On Windows, extract the archive for the machine's architecture and invoke nova.exe directly or place its directory on PATH. The Windows binaries are not Authenticode-signed, so SmartScreen may warn on first run. The macOS binary is ad-hoc signed, not notarized.

Published release assets never change. Every consumer pins these files by SHA-256 — Homebrew and npm formulas, Bamboo's plugin catalogue (bundle, manifest, per-platform executable and app archive digests), and any manual --sha256 install. Re-uploading a same-named asset to an existing release would silently break all of them, so the release workflow refuses to publish when the tag already has assets:

error: v0.3.1 was already published with 2 asset(s); refusing to replace them.

A re-cut ships a new version tag (bump the patch version) rather than replacing bytes under the old one. Deliberately replacing an existing release's assets requires scripts/verify-release-immutable.sh <tag> --allow-asset-replace, which warns that every pinned consumer is now broken. Because a tag's assets are immutable, an already-published release can no longer be re-run to add missing files: if a partial publish happened, ship the next patch version. Draft releases are exempt, since a draft is not published yet.

Build from source:

git clone https://github.com/bigduu/Nova.git
cd Nova
cargo build --release --locked

The result is target/release/nova on macOS or target/release/nova.exe on Windows. Do not use cargo install nova: that crates.io name belongs to an unrelated project. The npm package is @bigduu/nova (see above).

Related MCP server: mcp-macos-cua

Choose a version

Path

What you get

Published v0.3.0 (Homebrew and release archives)

Screenshots, zoom_region, OCR, numbered marks with click_mark, mouse/keyboard input, AX/UIA actions including ax_read / read_ui / ax_activate, inspect_app, managed nova mcp, Nova.app development preview, optional Chrome DevTools sidecar, consent-based Chrome bridge, windows/apps, clipboard, batch_actions (64-step limit + structured failure), wait.

master (tip of development)

Whatever lands after v0.3.0 — build from source for unreleased work.

Nova.app remains a development preview (ad-hoc signed, not notarized). Menu-bar status and virtual cursor are labelled Preview pending live-device acceptance (#34, #70). Audit evidence.

Use it from an MCP client

These snippets use direct stdio (no arguments). They work with the v0.3.0 release and with a master build. Replace the path with your own which nova output, the extracted binary or target/release/nova. GUI clients do not always inherit your shell PATH, so use an absolute path.

Claude Desktop: edit ~/Library/Application Support/Claude/claude_desktop_config.json on macOS or %APPDATA%\Claude\claude_desktop_config.json on Windows. Cursor: edit ~/.cursor/mcp.json (all projects) or .cursor/mcp.json in one project.

{
  "mcpServers": {
    "nova": { "command": "/opt/homebrew/bin/nova", "args": [] }
  }
}

Claude Code:

claude mcp add --scope user nova -- /opt/homebrew/bin/nova

Codex: add to ~/.codex/config.toml, or run codex mcp add nova -- /opt/homebrew/bin/nova:

[mcp_servers.nova]
command = "/opt/homebrew/bin/nova"
args = []

On Windows, use the full path to nova.exe, escaped in JSON ("C:\\Tools\\nova\\nova.exe") or as a TOML literal string ('C:\Tools\nova\nova.exe').

macOS permissions: in direct stdio mode, macOS usually attributes Nova to the app that launches it. Grant Accessibility (input and UI control) and Screen Recording (screenshot, ocr, list_windows) to Claude Desktop, Cursor, or the terminal/IDE that runs Claude Code or Codex. If that does not work, add the nova binary itself. Then restart or reconnect the MCP server. You can instead let Nova.app own the permissions; see Managed mode.

Try: "Use Nova to list my open windows, take a screenshot of the frontmost one and tell me which buttons you can see."

Managed mode with Nova.app

nova mcp is the cross-platform managed entrypoint used by the Bamboo plugin (v0.3.0+):

{
  "mcpServers": {
    "nova": { "command": "/absolute/path/to/nova", "args": ["mcp"] }
  }
}

Windows and Linux headless builds serve ordinary stdio MCP. On macOS, mcp only connects to the independent Nova.app, launching it through LaunchServices when needed. Install the app separately, open it once, and grant Accessibility to Nova; Screen Recording is needed for capture, OCR, and list_windows (the app menu requests it only when you explicitly choose that action). The bundled executable can also be used as the connector:

{
  "mcpServers": {
    "nova": {
      "command": "/Applications/Nova.app/Contents/MacOS/nova",
      "args": ["mcp"]
    }
  }
}

The explicit --connect command remains supported and uses the same transport as macOS mcp. It carries MCP bytes over a private per-user Unix socket. The connector does not call desktop APIs or request macOS permissions; the app process owns the MCP handlers and TCC responsibility. The socket lives under /tmp/nova-app-<uid>/ with a mode-0700 directory, mode-0600 socket, and a same-UID peer check.

If Nova.app is unavailable, the managed command exits with installation and reconnection guidance. It never falls back to desktop operations inside the MCP host. NOVA_APP_SOCKET is for isolated development/tests; when set it disables automatic app launch. Unset it for the normal installed-app setup. Unbundled nova with no arguments still offers the legacy direct stdio mode.

When the app service closes its connection, the CLI connector finishes forwarding responses and exits even if the host keeps its stdin pipe open. Closing host stdin normally still half-closes the request stream and drains the service's final response, including its last buffered bytes. Standard stdout backpressure still applies: the host must keep reading responses. Reconnect only the Nova MCP server after replacing/restarting the service; Bodhi can stay open. No interrupted request is replayed and no replacement MCP session is created automatically.

This exit behavior is specific to the terminating CLI connector process. Its dedicated runtime is released after forwarding completes, and process exit reclaims an outstanding blocking stdin read. It does not make the connect_stdio library function's stdin cancellable inside a resident or embedded runtime.

Chrome DevTools MCP sidecar

For advanced Chrome page automation and debugging, Nova can launch the official Chrome DevTools MCP next to the desktop server. This is a transparent stdio sidecar, not a second browser implementation inside Nova. It requires npm/npx, Node.js ^20.19.0, ^22.12.0, or >=23, and current stable Chrome (or newer). Nova pins the reviewed upstream package to chrome-devtools-mcp@1.8.0.

Application-level integrations should accept an app selector (name or bundle ID) and use inspect_app discovery internally. Discovery alone does not grant CDP control; automatic app-to-provider routing remains a separate integration slice. Endpoint copying and the separate MCP configuration below are advanced transport compatibility. On macOS:

{
  "mcpServers": {
    "nova": {
      "command": "/Applications/Nova.app/Contents/MacOS/nova",
      "args": ["--connect"]
    },
    "nova-chrome-devtools": {
      "command": "/Applications/Nova.app/Contents/MacOS/nova",
      "args": ["chrome-devtools"]
    }
  }
}

For a standalone binary from the v0.3.0 release or current source, use its absolute path and ["chrome-devtools"]. If a GUI client cannot find npx, add "--npx", "/absolute/path/to/npx" after the subcommand.

The default launches a new temporary, isolated Chrome profile. Usage statistics, package update checks, CrUX URL lookups, and sensitive network headers are disabled/redacted by default. Requests made by attached DevTools targets can be guarded by repeating --allowed-url-pattern, for example:

"args": [
  "chrome-devtools",
  "--allowed-url-pattern", "https://example.com/*",
  "--allowed-url-pattern", "https://*.example.net/*"
]

URL allow patterns require Chrome 149+. They apply only to DevTools targets while the MCP server is attached and are not a complete network sandbox; use an OS/VM sandbox when full network isolation is required, as described by the upstream security policy.

To work with an already running signed-in Chrome profile instead, first open chrome://inspect/#remote-debugging in Chrome and enable remote debugging, then configure:

"args": ["chrome-devtools", "--profile", "existing"]

Automatic connection requires Chrome 144+. If several Chrome profiles are active, Chrome chooses the profile it considers the default; select and verify the connected pages before acting.

WARNING

Existing-profile mode can inspect and control every open window in the selected Chrome profile, including authenticated pages. Enable it only for a trusted local MCP client, and disable remote debugging when finished.

An internal consumer or advanced caller can attach to an explicitly selected, already-running browser instead of relying on stable Chrome's default profile:

nova chrome-devtools --browser-url http://127.0.0.1:9222
nova chrome-devtools --ws-endpoint 'ws://[::1]:9222/devtools/browser/<id>'

Use one endpoint, without --profile (including explicit isolated) or --headless. --browser-url accepts HTTP(S) at the browser root, with an optional trailing /; --ws-endpoint accepts WS(S) at /devtools/browser/<id>, where the ID uses letters, digits, hyphen or underscore. Both require an original literal loopback IP and explicit port 1–65535. Hostnames, abbreviated/integer/hex IPs, credentials, query strings and fragments are rejected before starting npx. Nova forwards the selected address literally and adds no Chrome launch flags or automatic connection fallback. HTTPS/WSS use upstream's certificate checks.

These inputs are trusted local endpoints, not network confinement. Pinned Puppeteer HTTP discovery uses the returned webSocketDebuggerUrl, and its WebSocket transport follows redirects. A local service can therefore lead the connection to another address. The pinned options and ownership cleanup distinguish attachment from launch: disconnect, stdin EOF or SIGTERM detach from an attached browser, while upstream closes a browser it launched itself.

After connecting, call list_pages and choose by the expected title/URL. Use the returned pageId explicitly on every page-scoped read/action. For two windows, if the intended pages have IDs 7 and 12, first take_snapshot({pageId: 7}) and fill({pageId: 7, uid: "<field from that snapshot>", value: "example"}); then take_snapshot({pageId: 12}) and click({pageId: 12, uid: "<button from that snapshot>"}). Obtain fresh page IDs and element UIDs after pages close or change; do not rely on whichever window is focused.

The endpoint must expose browser-level CDP, covering the app's pages/windows. The WS path check excludes renderer-only /devtools/page/... sockets and Node/V8 main-process inspector WebSocket URLs. HTTP root syntax alone cannot establish the advertised service's identity. Upstream officially targets Chrome/Chrome for Testing; Electron/CEF attachment is experimental and unverified for each runtime until reads/actions are tested in its own windows. A successful MCP tools/list alone does not establish application compatibility. On setup or connection failure, check the selected app's supported debugging setup and refresh its discovery metadata before retrying this transport. Keep Bodhi open; Nova's native AX and separately paired extension capabilities remain available.

Use --enable-webmcp to expose upstream's experimental WebMCP tools. Nova adds Chrome's required --enable-features=WebMCP launch argument in isolated mode; for an existing profile or endpoint, the browser must already have started with that feature enabled. WebMCP requires Chrome 150+. --expose-network-headers and --enable-performance-crux are explicit privacy opt-ins. The pinned 1.8.0 package does not support a --disable-javascript-evaluation option, so Nova does not advertise or pass it.

The sidecar and Nova's optional Secure Chrome Bridge serve different trust models: DevTools MCP is the broad, full-featured choice for normal browser automation, DOM/network inspection, and performance debugging; the Secure Chrome Bridge requires explicit per-page pairing and is preferable when least-privilege page scoping matters. Nova's desktop tools remain the path for browser chrome, native dialogs, and non-web UI.

Use the absolute path to the extracted release binary or the source-build output. On Windows, use an escaped executable path such as "C:\\absolute\\path\\nova.exe". Use the source-build output for the AX-first workflow below. If its directory is already on PATH, the command can be "nova".

Reconnect/reload the Nova MCP server in the client; Bodhi's main window can stay open. See Permissions & code signing for legacy direct-stdio and development-binary cases.

HTTP clients — run Nova as a server and connect over Streamable HTTP:

nova --http                       # 127.0.0.1:3100/mcp
nova --http --addr 127.0.0.1:8080 # custom loopback port

HTTP mode is currently a local transport: it keeps rmcp's default loopback Host allowlist and does not configure remote-access authentication. Binding all interfaces is not a supported LAN setup.

First calls (current source build). Call ax_read (optionally ax_read(window="<name>", mode="all")) for semantic content and controls, then ax_activate(snapshot_id, node_id) on an exact actionable node. Re-run ax_read after the action to verify semantic state. If AX/UIA coverage is absent or partial, use focused-window ocr for rendered text; use screenshot(window=...) / zoom_region only when pixels are necessary (layout, icon, color, image, canvas, or visual verification). All pointer tools use the pixel space of the most recent screenshot; cursor_position instead reports OS-global logical coordinates.

Semantic reads (ax_read and its read_ui alias) include supported control state and identify scroll containers with scrollable=true in all/content mode, including unnamed containers. Containers alone receive no action mark. The target includes its window title and, when the selected macOS window exposes AXDocument/AXURL, an optional url. Unsupported URL/state attributes are omitted; Windows uses supported cached UIA states and ScrollPattern metadata.

Windows reads preserve cached ValuePattern text, including Unicode and multiline values, for controls that expose the pattern. Empty, unsupported, failed, or non-string values are absent; password fields and unknown password status stay redacted. Controls exposing only TextPattern have no value fallback.

Native text writes (type_text, write_clipboard, batch typing, and ax_set_value) omit the submitted text from tracing and acknowledgements. Write diagnostics retain character/UTF-8 byte counts and native operation details without echoing the value. Explicit reads such as read_clipboard and ax_read still return the requested content.

Requirements

  • Linux: the source builds a headless MCP server for protocol checks; native desktop actions return an unsupported/headless error. This is not Linux desktop automation.

  • macOS 14+ for the macOS desktop backend. The release archive is universal and runs on Apple Silicon and Intel Macs.

  • Windows x86_64 or ARM64 for the Windows desktop backend. GitHub Releases provide a native archive for each architecture.

  • On Windows, ocr uses installed Windows OCR language packs. Use nova --ocr-langs to inspect available languages; install the needed pack if recognition reports that it is unavailable.

  • Building on macOS requires the macOS 15 SDK / Xcode 16+ because of a transitive apple-metal build dependency; that is a build-time requirement, not Nova's minimum macOS runtime version.

  • On macOS, Screen Recording permission is required for screenshot, ocr, and list_windows; Accessibility is required for ax_read, semantic activation, and input.

macOS grants these permissions to the process it identifies as responsible for Nova. The managed nova mcp entrypoint and Bamboo plugin use the independent Nova.app on macOS. Direct stdio/HTTP can use the host app, terminal, or directly launched binary as the permission subject. See Permissions & code signing.

Tools

Tool

What it does

ax_read

Canonical ax:read: read semantic labels, text, values, roles, actions, state, and optional bounds through macOS Accessibility or Windows UIA, without a screenshot. Returns an ephemeral snapshot/node protocol and explicit coverage/status.

read_ui

Compatibility alias backed by the same ax_read traversal and cache generation.

ax_activate

Activate an exact actionable node from a fresh ax_read; rejects stale snapshot IDs and reports route=ax|uia|web_dom|element_center. Every attempt consumes its generation before provider dispatch.

screenshot

Capture the whole display or a single window= — use for layout, icons, colors, images, canvas, and visual verification after semantic/OCR paths.

zoom_region

Magnify a rectangle of the last screenshot at native resolution — reads small targets on surfaces with no Accessibility tree.

ocr

Recognize on-screen text with Apple Vision on macOS or Windows Media OCR on Windows. mode=auto uses Fast first with confidence-based Accurate fallback; mode=fast|accurate forces either policy. An optional strict roi={x,y,width,height} re-captures a rectangle from the current image through the native region path. Returns each line with a clickable center.

click_mark

Compatibility action for the latest numbered mark; prefer generation-safe ax_activate.

left_click / right_click / double_click / mouse_move / scroll

Pointer input in the pixel space of the last screenshot.

cursor_position

Read the cursor in OS-global logical coordinates; it is not converted into the last screenshot's pixel space.

type_text / key_combo

Keyboard input (full Unicode, incl. CJK + emoji).

list_windows / list_applications / open_application

Window & app introspection.

inspect_app

Optional macOS app capability discovery. Accepts an app name/bundle ID, or discovers running Chromium candidates when omitted; no caller-supplied port or permission prompt.

read_clipboard / write_clipboard

Clipboard access.

ax_click / ax_set_value / ax_focus

Drive controls by Accessibility role/label.

dump_ax

Read the raw AX/UIA tree for diagnostics and coverage debugging.

batch_actions

Run a sequence of input actions in one call.

wait

Pause for a specified number of seconds.

batch_actions runs at most 64 actions sequentially and stops at the first failure, without retry or rollback. Success keeps the newline-separated status lines. Failure sets MCP isError=true and returns the same JSON in text content and structuredContent: completed: [{index, result}], failed_index, reason, and not_executed: {start, end_exclusive}. Indices are zero-based; the half-open range includes only actions after the failed step, which may already have partial side effects. More than 64 actions is rejected before any action runs, with failed_index: null and the whole batch marked unexecuted. Each acknowledgement or reason is limited to 512 Unicode characters, retaining its beginning and end with … in the middle when shortened. Typed text keeps character/UTF-8 byte counts rather than plaintext. Inspect progress and obtain fresh ax_read state before deciding what to retry; replaying the whole batch could repeat completed side effects.

Run

cargo run                      # stdio transport (default)
cargo run -- mcp                # managed MCP: Nova.app on macOS, stdio elsewhere
cargo run -- --http            # Streamable HTTP on 127.0.0.1:3100
cargo run -- --http --addr 127.0.0.1:8080

The Swift runtime that ScreenCaptureKit links is located via an LC_RPATH baked in by build.rs, so no DYLD_* environment variable is needed for cargo run/cargo test or the standalone binary.

Nova.app development preview

Releases cut from a revision containing the app packaging workflow also attach:

nova-v<version>-universal-apple-darwin-development-app.zip

This archive contains a universal Nova.app that runs Nova's per-user app service without a Dock icon. It gives Screen Recording and Accessibility a Nova application identity instead of making the MCP host (for example, Bodhi) the permission subject. Install and start it with:

shasum -a 256 -c nova-v*-universal-apple-darwin-development-app.zip.sha256
unzip nova-v*-universal-apple-darwin-development-app.zip
ditto Nova.app /Applications/Nova.app
open -gj -b com.zenith.nova

Install the app independently of Bodhi and the plugin's downloaded CLI. Keep it at /Applications/Nova.app (or ~/Applications/Nova.app), outside Bodhi.app and the plugin directory. The plugin still downloads the CLI archive and uses it only as the connector on macOS; installing/updating the plugin does not install or update Nova.app. Use a CLI and app built from the same current version.

Configure a stdio MCP client with nova mcp, as shown in Use it from an MCP client below. If no app archive has been published for the current code, build both macOS architectures, combine them into a universal binary, then assemble the app with package-development-app.sh, which requires a universal binary and the matching Cargo version as arguments.

WARNING

The app archive isDEVELOPMENT ONLY. It is ad-hoc signed, not Developer ID signed, not notarized, and not stapled. Gatekeeper can block it, and replacing it with a differently signed build can require granting TCC permissions again. The existing universal CLI .tar.gz remains the supported artifact consumed by Homebrew and Bamboo; the app .zip does not replace it.

Initial app setup and explicit local signing (0.3.1 source)

The current source adds nova app status and an explicit initial installer:

nova app status
nova app install --archive /absolute/path/to/nova-v0.3.1-universal-apple-darwin-development-app.zip \
  --sha256 <64-hex-digits-from-a-trusted-release-or-catalogue>

These commands are not included in the already published v0.3.0 connector. Build this source or use the matching 0.3.1 release when published; installing the existing v0.3.0 Bamboo plugin does not invoke setup or install Nova.app.

Both commands print one JSON report with state, success, message, and next_action; valid apps also report their path, version, signing type, development, and connector_compatible. Status checks the current account's ~/Applications/Nova.app, then /Applications/Nova.app, without creating directories, starting the app, capturing the desktop, or requesting permissions. States are missing, installed, already_installed, unsupported, and error; refused or unsupported installation exits unsuccessfully.

Installation accepts a local ZIP and independently supplied SHA256. It checks the archive, bundle identity, both Mac CPU architectures and code signature, and requires the app version to match this connector. It publishes only an absent ~/Applications/Nova.app, refusing any existing canonical app, symlink, or unsafe installation directory. Source ZIP bytes stay unchanged. By default, app metadata and download quarantine are retained, and no signing identity is accessed. SHA256 verifies the selected bytes; obtain it from a trusted publisher. There is no replacement or update command, automatic launch, or privacy grant.

For the explicitly selected local_ad_hoc distribution path, add --local-sign to the same install command. After hash, ZIP, app identity, architecture, original signature and version checks pass, setup removes only the private staged app's download quarantine and ad-hoc signs its executable and app, inside out, with the fixed com.zenith.nova identifier. It then verifies the signatures before initial publication. The supported app's only executable is Contents/MacOS/nova; unexpected nested executables are rejected.

This mode never accesses a keychain, creates a certificate, changes certificate trust, or grants privacy permissions. Signing or quarantine errors return an error JSON report with manual guidance and clean up the private stage. The original archive and any existing canonical app remain unchanged.

Both the original development archive and a local_ad_hoc installation report signing: "ad_hoc" and development: true. Neither is Developer ID signed or Apple-notarized. Ad-hoc signatures depend on the app's bytes: the fixed bundle identifier does not establish TCC continuity across app updates. Status stays passive. Open Nova.app through macOS and grant the permissions you need, then reconnect only Nova MCP; Bodhi can remain open. Existing apps, including /Applications/Nova.app, remain untouched even with --local-sign. Production Apple signing, upgrades and real TCC acceptance remain separate work tracked in #36.

Permissions & code signing (macOS)

Inspect an application's interaction options

Use inspect_app when setting up an application or checking which interaction route is available. It is optional; ordinary native interaction still starts with ax_read.

{"app": "Slack"}

The selector accepts a running application's name or bundle identifier. Exact matches take priority over partial matches. Omit app to discover running Electron, Chromium, and CEF candidates, including applications with no discovered debugging connection. Names alone do not confirm a runtime: Nova checks known framework containers and their executable evidence. Unknown or unreadable bundles remain unknown.

The default result contains application identity, runtime, inspection status, the currently available native route, and a next step. Nova finds process-owned local connection candidates internally; callers do not need to find or supply ports. For diagnostics only, use:

{"app": "com.example.application", "details": true}

Detailed output includes bundle/runtime evidence, process start identities, endpoint provenance, and metadata verification. Nova checks the selected app's owned listeners, recognized debugging flags, and the exact DevToolsActivePort file only when a --user-data-dir flag evidences the profile location. It does not scan profile contents or return full arguments/environment. Programmatically enabled ports can be discovered through listener ownership even when a flag is absent from the OS argument list.

browser_endpoint_available means a metadata-only browser handshake succeeded; it does not attach browser tools, grant authorization, or verify the full Chrome DevTools MCP toolset. Native ax_read still uses Accessibility, and the result reports when that permission is needed. Node inspector endpoints, incompatible endpoints, stale evidence, and incomplete inspection remain distinct. No discovered port is not proof that debugging is disabled. Enablement and whether a particular application can support a restart-based change remain unknown until verified for that application.

Discovery does not launch, focus, quit, or restart applications, request permissions, modify bundles/arguments, or open a debugging service. Network requests stay on verified process-owned loopback sockets: /json/version, Browser.getVersion, and Target.getBrowserContexts only. There is no page enumeration, script evaluation, input, or Browser.close. HTTP proxies and redirects are disabled; advertised WebSockets must keep the same owned address and port. Ownership/start identity is checked before and after probing.

An investigation allows 8 seconds overall, 16 result apps, 32 processes per app, 4 helper generations, 8 endpoint probes per app, and 2 evidenced profiles. Each metadata probe has a 900 ms deadline; HTTP bodies and WebSocket messages are limited to 32 KiB, the WebSocket exchange to 128 KiB and 16 frames per reply. Framework lookup is limited to 64 entries in an app's Contents/Frameworks, plus at most four version directories in each recognized framework. Limits or unavailable evidence are reported as incomplete, rather than silently claiming that an application has no debugging support. Concurrent calls receive a busy result. Windows/Linux return an explicit unsupported result; their existing native tools are unchanged.

On macOS, resident desktop transports keep the process main run loop active, so applications launched or quit after the first inspection appear or disappear without restarting Nova or its MCP host. Relaunching an application triggers fresh process/start-time and endpoint ownership checks. This inventory refresh does not require Screen Recording or Accessibility permission. The mcp and --connect byte proxies return before this desktop event loop and bootstrap.

The automated tests use fake bundles, process records, and loopback services. The ignored own_listener_and_process_start_identity_match test inspects only its own process/listener. The ignored e2e_app_inspection acceptance test requires an explicitly prepared app with a dev.nova.acceptance.* bundle identifier and NOVA_TEST_APP_BUNDLE_ID; it never defaults to inspecting the user's running applications.

cargo test --test e2e_resident_app_inspection runs a separate macOS regression whose test binary owns the real process main thread. In one resident process it seeds discovery, launches a unique temporary AppKit app, checks appearance, quits it, checks disappearance, and checks a new process identity on relaunch. It repeats this with an internally allocated random listener that simulates the narrow CDP handshake; the fixture is not Chromium and is reported as an unknown runtime. It creates no windows and requests no permissions. Real Electron/Chromium lifecycle acceptance remains a separate check. The test-only --without-main-loop argument is a negative control that reproduces the old stale-inventory failure; it is expected to fail.

Permission ownership

The Nova.app menu is a Preview in v0.3.0; real-desktop acceptance is tracked in #34. The packaged macOS app has a Nova menu-bar entry. It shows the local service's Starting, Ready, or Failed state separately from Accessibility and Screen Recording. Ready means the local service is listening; it does not imply either permission is granted or Chrome is paired. If startup fails, the menu stays available with a failed state. A duplicate launch exits and leaves the existing service owner running.

Startup and Refresh Status only check current permissions. They do not capture anything, open Settings, request a permission, or wait on the Chrome pairing queue. Request Accessibility… asks only for reading and operating native application controls. Request Screen Recording… asks only for screenshots and screen text recognition. Each section also has its own Open … Settings action. A “Not granted” result may mean permission has not been requested yet; it does not distinguish that case from denial.

After changing a permission, choose Refresh Status and retry the Nova tool; Bodhi can remain open. Capture continues to use the existing helper and its permission-change recovery on the next capture request; refresh itself neither starts nor restarts a capture helper. Quit Nova ends the service and its connections. Reopen Nova and reconnect only the Nova MCP server in the client when needed. There is no automatic request replay or restart control.

The menu is available only in the macOS app service. Direct stdio/HTTP and connector processes keep their existing transports and have no status menu. This UI does not guarantee that permissions survive replacing/signing Nova or an OS update; those remain separate installation and release checks.

The independent app transport is the preferred permission model: grant Screen Recording and Accessibility to Nova.app, then use nova mcp (or explicit nova --connect). The connector never initializes CoreGraphics or Accessibility, so Bamboo, Claude Desktop, and terminals no longer need Nova's desktop permissions.

Keep Nova.app installed independently and unchanged when upgrading Bodhi. The new Bodhi/plugin connector connects to the same app-owned service, so its own build/signing identity does not become Nova's permission subject. This is an architectural guarantee about where desktop calls execute; signed installation and real TCC upgrade acceptance remain separate release gates. Replacing Nova.app itself, changing its signature, or an OS permission decision can still require granting permissions again. The development preview is ad-hoc signed.

After granting Nova permissions in System Settings, retry the tool. If macOS requires a restart for the change, quit/reopen Nova.app, then reconnect only the Nova MCP server in the client. Keep Bodhi's main window open. The connector does not replay interrupted requests or automatically restore an MCP session after Nova exits. Do not remove/re-add Bodhi's grants to repair this managed Nova path.

Two details still matter for direct stdio/HTTP and source-development modes:

Grant the responsible process for the way Nova is launched. macOS TCC may attribute a child process to its responsible parent app. For legacy direct stdio MCP (an empty argument list), grant Claude Desktop, Bamboo, or the terminal/IDE that launches Nova. For a directly launched CLI/HTTP process, macOS may instead use the Nova binary. If granting the expected host does not work, add the installed nova binary (or target/release/nova) as a fallback under System Settings → Privacy & Security → Screen Recording and Accessibility.

Keep the identity of whichever process receives the grant stable. If Nova itself is the permission subject, cargo build produces an ad-hoc, linker-signed binary whose code-signing identity is a content hash (nova-<hash>). It changes every build, so a direct binary grant stops applying. Sign Nova with a stable self-signed identity when developing in that mode:

cargo build --release
./scripts/dev-codesign.sh --release   # re-sign after EVERY build

The first run creates a Zenith Nova Code Signing identity in your login keychain (click Always Allow once if codesign prompts) and signs the binary with a fixed identifier (com.zenith.nova). A direct Nova grant then survives rebuilds that are re-signed with the same certificate. Host-app grants likewise depend on the host keeping a stable signing identity.

Troubleshooting — screenshot fails with a "wedged" / "busy" capture error. All captures (and window enumeration) run in ONE shared per-user daemon (nova --capture-daemon, flock-elected, socket /tmp/nova-capture-<uid>-<hash>.sock), because replayd keys clients by executable path — two same-binary ScreenCaptureKit clients evict each other's XPC identity and wedge every new stream start. The daemon kills itself if a capture exceeds its 8s watchdog, and the client auto-recovers: kill+respawn the daemon, then (second failure) SIGKILL all nova capture processes and killall -9 replayd — wedges self-heal without manual action. If they don't: nova --selftest (probes ScreenCaptureKit in a sacrificial subprocess, then the daemon path) and read /tmp/nova-capture-worker.log (step trace) + /tmp/nova-capture-daemon.log (daemon stderr). Manual remedy = kill the processes holding streams (pkill -f -- --capture-daemon), NOT replayd: plain killall replayd is a no-op (replayd ignores SIGTERM), and even killall -9 replayd doesn't cure a wedge while a stream-holding client survives — it just reconnects and re-wedges the fresh replayd.

Coordinate grounding

The virtual cursor below is part of the Nova.app Preview in v0.3.0; real-desktop acceptance is tracked in #70. In macOS Nova.app, coordinate-backed mouse movement, clicks and scrolls also show a purple virtual arrow at the supplied logical point. Click rings and scroll direction cues fade after 400 ms; the arrow clears after 1.2 seconds of inactivity. The panel passes through mouse input and cannot take keyboard focus. Foreground input still moves the real pointer; PID/background delivery keeps its existing behavior. Batch uses the same native backend. Internal pointer restoration after element-center fallback leaves the cue at that click. Feedback marks an input attempt; it does not confirm the target accepted it. Semantic-only AX actions and browser DOM actions do not infer cursor positions; direct no-UI transports and pure connectors do not create this overlay.

Display and region captures exclude the reserved cursor window only when its owner resolves to the same Nova executable, including updates to a warm capture stream. Other-app single-window captures keep their existing filter. Rendering, click-through, focus/pointer behavior, batch, cold/warm screenshot exclusion and Quit cleanup still require controlled real desktop acceptance; automated tests do not establish those GUI results. Multi-display and full-screen behavior also need live verification on the available setup.

A general LLM judging pixel coordinates off a downscaled screenshot is the main source of mis-clicks — so the primary path avoids pixels entirely.

  • ax_read first (no image) — returns actionable controls and non-actionable readable content in deterministic tree order. A successful macOS read requires Accessibility but does not contact ScreenCaptureKit. permission_denied means fix that grant; it is not an instruction to take a screenshot.

  • Fresh semantic action — call ax_activate with the returned snapshot and node IDs. Native AX/UIA and the browser DOM bridge are tried before a freshly revalidated element-center click. Stale generations fail closed; every activation attempt consumes its generation before provider dispatch, so read again after any result.

  • OCR second — when coverage is absent/partial and the missing information is rendered text, use focused-window OCR and its returned text center with left_click(..., source="ocr_center").

  • Screenshot/zoom last — use pixels for visual-only state or a surface with no semantic/text representation; coordinate clicks report route=visual_coordinate. Screenshot marks and click_mark remain available for compatibility.

When a screenshot is needed, all click/move/scroll tools work in the pixel space of the last screenshot — the server remembers that frame and maps clicks back to the real screen, so the model just "clicks what it sees":

  • screenshot(window: "<name>") — capture a single window (substring of its title or app name) instead of the whole display. Smaller, sharper image → less context and far less downscaling → better precision. Later clicks map into that window.

  • zoom_region(x, y, w, h) — magnify a rectangle of the last screenshot at native resolution (capturing only that rectangle). For reading small targets on surfaces that expose no Accessibility tree (canvas, games, custom views), where coordinates are the only option. A labeled coordinate grid is overlaid so the model reads positions straight off the axes.

Testing

The suite is split into fast, hermetic tests (run by default) and side-effecting end-to-end tests (opt-in, #[ignore]d).

Default — unit + hermetic integration tests

cargo test

Runs everything that has no side effects and needs no special permission:

  • unit tests for coordinate scaling, the key/char keystroke maps, combo parsing, batch (de)serialization, and MCP tool registration;

  • tests/e2e_interaction.rs — screenshot→logical coordinate mapping (via CGDisplay, no permission needed) and a non-destructive clipboard round-trip (snapshots and restores the clipboard).

This is what the macOS test job runs in CI (see .github/workflows/ci.yml; the workflow also has Windows cross-check and Linux headless jobs).

End-to-end tests (#[ignore]d)

These either post real input events (they move the cursor, click, scroll, or type into the focused window) or require Screen Recording permission, so they are excluded from cargo test and must be opted into. Run them on a desktop session where that's acceptable:

# all of them
cargo test -- --include-ignored

# or a single one
cargo test --test e2e_input mouse_move_roundtrips_through_cursor_position -- --ignored

Test (file)

What it does

Needs

semantic_snapshot_reads… (e2e_ax_read)

Resolves and reads a focused or NOVA_AX_WINDOW app through AX/UIA without pixel capture

Accessibility / logged-in UIA desktop

mouse_move_roundtrips… (e2e_input)

Moves the cursor, reads it back via cursor_position, asserts the position — restores the cursor

Accessibility

click_events_post… (e2e_input)

Left/right/double click on the empty desktop corner (Esc dismisses the menu)

Accessibility

scroll_events_post… (e2e_input)

Posts vertical scroll events

Accessibility

type_text_posts… (e2e_input)

Types into the focused window

Accessibility

open_application_launches… (e2e_input)

Launches/focuses System Settings

—

list_windows_returns… (e2e_input)

Enumerates on-screen windows

Screen Recording

e2e_capture_display_returns_valid_jpeg (e2e_screenshot)

Captures the display, checks the JPEG

Screen Recording

e2e_capture_dims_match_target_dims_contract (e2e_screenshot)

Asserts capture dims match the click-coordinate mapping

Screen Recording

e2e_window_screenshot_produces_view_frame (e2e_screenshot)

Captures a window and validates its view-frame metadata

Screen Recording

ocr_recognizes_text_on_the_display (e2e_ocr)

Runs Apple Vision OCR on a live capture; asserts text + in-bounds line centers

Screen Recording

daemon_* / client_* / concurrent_* (e2e_capture_worker)

Shared capture daemon: capture, kill→respawn recovery, concurrent clients, clean-error survival

Screen Recording

legacy_pipe_protocol_still_served (e2e_worker)

Old --capture-worker pipe protocol, proxied into the daemon

Screen Recording

stdio_server_completes_handshake_and_lists_tools (e2e_stdio)

Exercises the stdio (JSON-RPC) transport end-to-end

—

safari_opens_google_and_nova_reads_the_homepage (e2e_safari_google)

Launches Safari, opens Google, and reads the page through Nova

Network + Screen Recording + Accessibility

mouse_move_roundtrips… proves the macOS pointer post and cursor read-back round-trip using logical coordinates. The non-ignored interaction test covers screenshot→logical coordinate arithmetic, while the live screenshot tests cover captured-dimension contracts.

Run e2e_capture_worker single-threaded (-- --ignored --test-threads=1): the tests share one daemon/socket.

list_applications_returns_app_bundles (in e2e_input) is not ignored — it only reads Spotlight and is tolerant of a Spotlight-less CI host.

Lint & format

cargo fmt --all -- --check
cargo clippy --all-targets

Releasing (maintainers)

A version tag drives everything via .github/workflows/release.yml. The workflow resolves the tag once, verifies it against Cargo.toml and the event commit, and makes every source-building job check out that immutable commit. It builds and smoke-tests the universal macOS CLI and development-only Nova.app, creates the Release with those assets, then sequenced jobs attach native Windows x86_64/ARM64 archives and the Bamboo plugin bundle. The CLI .tar.gz name and checksum outputs stay unchanged for Homebrew and the Bamboo plugin manifest.

Run the hermetic release checks before tagging:

scripts/test-release-workflow.sh

The published release is v0.3.0. When cutting the next release, make sure the tag matches the manifest version. Release tags must be protected from force updates; the workflow also serializes runs by tag and re-verifies the tag before its first upload. The Nova.app asset must remain labeled DEVELOPMENT ONLY until all production distribution gates are complete:

  • sign nested code and the outer app, in that order, with a Developer ID Application identity and the hardened runtime;

  • submit the distribution artifact to Apple's notary service and verify the accepted ticket;

  • staple the ticket to the app and validate it with codesign and spctl;

  • authenticate local MCP and Chrome bridge peers with macOS audit tokens and designated code requirements, rather than relying on same-UID sockets alone;

  • run the packaged native host and extension against a real Chrome install, including pairing, navigation revocation, stale snapshots, and disconnects;

  • pin third-party GitHub Actions by full commit SHA before treating the release workflow as a production supply-chain boundary;

  • smoke-test launch, upgrade, nova --connect, Screen Recording, and Accessibility grants on clean Apple Silicon and Intel macOS 14+ machines.

Do not describe the ad-hoc-signed app preview as a production-ready macOS app.

License

MIT © bigduu

Available Tools

35 tools
ax_activateA

Activate one actionable node from the immediately preceding ax_read using its snapshot_id and node_id. This generation-safe action fails closed when the UI has been read again. Paired-page nodes use their captured Chrome Session/route and provider IDs; DOM errors never use native/coordinate fallback. Native nodes use the web DOM bridge for scriptable browser content, native AX/UIA activation otherwise, then a freshly verified element-center fallback. The result reports route=chrome_extension|ax|uia|web_dom|element_center. An accepted current actionable token consumes that generation before provider dispatch, so rerun ax_read after any result.

Inputs: node_id (required), snapshot_id (required), background.

ax_clickA

Legacy query action through macOS Accessibility or Windows UI Automation — no coordinates, no cursor movement, works in the background (the app need not be frontmost). query is a case-insensitive substring of the element's accessibility role or label/title (e.g. "Send", "Search"). Ambiguous matches fail closed. Prefer fresh ax_read + ax_activate for exact, generation-safe actions.

Inputs: query (required).

ax_focusA

Legacy query action that focuses a control through macOS Accessibility or Windows UI Automation. Background, no cursor. Ambiguous substring matches fail closed; prefer a fresh ax_read node whenever possible.

Inputs: query (required).

ax_readA

Canonical ax:read capability. target=paired_page selects only the already consented Chrome document, before native targeting; mutually exclusive with window. Page nodes have provider=chrome_extension and no native marks/global bounds; page failures require inspection/re-pairing and never native or pixel fallback. target=native (default) reads visible semantic UI directly from macOS Accessibility or Windows UI Automation without taking a screenshot. Returns an ephemeral snapshot_id, deterministic node ids, role/name/description/value/actions/state, optional global-logical bounds, explicit coverage/fallback status, and actionable marks. mode=all (default) combines controls with labels/static text/headings; interactive or content narrows the view. Call ax_activate(snapshot_id,node_id) on a fresh actionable node. If coverage is absent/partial, use focused OCR for missing rendered text, then screenshot/zoom only for visual-only pixels. permission_denied means grant Accessibility; do not hide it with a screenshot fallback.

Inputs: max, mode, filter, target, +2 more.

ax_set_valueA

Legacy query action that sets a control value through macOS Accessibility or Windows UI Automation. Background, no cursor. Ambiguous substring matches fail closed; prefer a fresh ax_read node whenever possible.

Inputs: query (required), value (required).

batch_actionsA

Execute a sequence of input actions (mouse_move, left_click, right_click, double_click, scroll, key_combo, type_text, wait) in one call to reduce round-trips, at most 64 actions. Coordinates are in screenshot space. Stops at the first failure and returns completed indices/results, failed_index/reason, and a not_executed half-open range. The failed action may have partial side effects. No retry or rollback. Get fresh ax_read state before deciding what to retry; use a screenshot for visual outcomes.

Inputs: actions (required).

chrome_activateA

Semantically activate one exact actionable node from the immediately preceding chrome_read snapshot. Fails closed on stale snapshot, route, capability, or DOM identity; coordinate fallback is forbidden.

Inputs: node_id (required), snapshot_id (required).

chrome_focusA

Focus one exact focus-capable node from the immediately preceding chrome_read snapshot. Fails closed instead of guessing or clicking.

Inputs: node_id (required), snapshot_id (required).

chrome_pairA

Begin an explicit 30-second pairing request for the active Chrome page. The user must inspect the origin and click Pair in the Nova extension popup. Success binds the exact tab, documentId, page nonce, and new epoch; it never grants access to another tab or later navigation.

Inputs: no named parameters.

chrome_readA

Read a bounded semantic DOM snapshot from the exact paired Chrome document. Sensitive fields and invalid/presentation ARIA nodes are excluded. Returns an ephemeral snapshotId and exact nodeIds; run this immediately before every Chrome mutation. No screenshot or coordinates are used.

Inputs: max_chars, max_nodes.

chrome_releaseB

Explicitly release the currently paired Chrome document, revoke its epoch, in-flight requests, and unexpired receipts.

Inputs: no named parameters.

chrome_scrollB

Semantically scroll one exact scroll-capable node (including root when exposed) from the immediately preceding chrome_read snapshot. Direction is up/down/left/right and amount is line/half_page/page; no coordinates.

Inputs: amount (required), node_id (required), direction (required), snapshot_id (required).

chrome_set_valueA

Set one exact non-sensitive text control from the immediately preceding chrome_read snapshot. Password, payment, OTP, file, hidden, and explicitly sensitive controls are unavailable. Plaintext is never logged or echoed; success returns only UTF-8 byte length and SHA-256.

Inputs: value (required), node_id (required), snapshot_id (required).

chrome_statusA

Report whether Nova's Chrome Native Messaging host is connected and whether one exact top-level document is currently paired. Call this before Chrome semantic work; it never inspects page content.

Inputs: no named parameters.

click_markA

Compatibility activation by the mark NUMBER shown by the most recent ax_read/read_ui or screenshot(marks=true). Prefer ax_activate with snapshot_id/node_id because it rejects stale generations. Web-page content in a scriptable browser (Safari, Chrome, Arc, Edge, Brave, …) is clicked through the page's OWN JavaScript engine (an Accessibility press is a silent no-op on web content), native controls through the Accessibility tree; if neither applies it falls back to a click at the element's center. Numbers go stale when the UI changes, so run a fresh ax_read immediately before acting.

Inputs: number (required), background.

cursor_positionA

Get the current mouse cursor position. Returns (x, y) in logical coordinates.

Inputs: no named parameters.

double_clickA

Grounded double click in the most recent capture's pixel space. Pass source=ocr_center for an OCR-returned center; otherwise the reported route defaults to visual_coordinate.

Inputs: x (required), y (required), source, background.

dump_axA

DEBUG: dump the target app's Accessibility tree (roles, subroles, labels, actions, frames) as indented text — to diagnose why some elements are not exposed. Targets the current AX/UIA app (or frontmost).

Inputs: no named parameters.

inspect_appA

Inspect a running macOS application's interaction options, or discover running Chromium applications when app is omitted. Returns a concise identity, status, native route and next step; details=true adds diagnostics. Read-only: no app launch, focus, permission request, page content, or browser attachment. Use optionally for setup/discovery; ordinary native interaction still starts with ax_read.

Inputs: app, details.

key_comboC

Simulate a key combination (e.g., "cmd+c", "shift+tab").

Inputs: key (required), background.

left_clickA

Last-resort grounded left click in the most recent capture's pixel space. Pass source=ocr_center when x/y came from focused OCR; otherwise source=visual_coordinate (the default) means a focused screenshot/zoom. The result reports that route.

Inputs: x (required), y (required), source, background.

list_applicationsA

List all installed applications on the system.

Inputs: no named parameters.

list_windowsA

List all visible windows across all applications.

Inputs: no named parameters.

mouse_moveB

Move the mouse cursor to the given (x, y) coordinates (in screenshot space).

Inputs: x (required), y (required).

ocrA

Read on-screen TEXT via the platform OCR engine (Apple Vision on macOS, Windows.Media.Ocr on Windows). Captures the display (or window="") and returns the recognized text lines, each with a clickable center in the same pixel space as a screenshot — so you can both READ the text and click a line with left_click(x, y). Returns text only (no image), so it is a cheap, fast way to pull text off the screen. Best when use it after ax_read reports absent/partial coverage and you need rendered TEXT from a surface where semantic nodes are empty or sparse — canvas, games, image-rendered or custom-drawn views — or to grab a lot of text at once without parsing a screenshot. For native/web UI with an Accessibility/UIA tree, ax_read + ax_activate is still more precise. mode=auto (default) tries fast recognition first and only falls back to accurate for empty/low-confidence output; mode=fast minimizes latency and mode=accurate forces maximum quality. roi={x,y,width,height} scopes OCR in the CURRENT image's pixels: the rectangle is strictly validated, mapped to the desktop, and re-captured through the native region path (never cropped from a downscaled JPEG). roi and window are mutually exclusive. Languages default to Simplified Chinese + English; pass languages=[...] (BCP-47) to override.

Inputs: roi, mode, window, languages.

open_applicationA

Launch or focus an application by name (e.g., "Safari", "Slack").

Inputs: app (required).

read_clipboardA

Read the current system clipboard contents as text.

Inputs: no named parameters.

read_uiA

Compatibility alias for ax_read, backed by the exact same semantic traversal, renderer, snapshot generation, and action cache. Prefer the canonical ax_read name in new prompts and integrations.

Inputs: max, mode, filter, target, +2 more.

right_clickB

Grounded right click in the most recent capture's pixel space. Pass source=ocr_center for an OCR-returned center; otherwise the reported route defaults to visual_coordinate.

Inputs: x (required), y (required), source, background.

screenshotA

Visual fallback after ax_read and, for missing rendered text, OCR. Capture the whole main display or a single window with window="" — and return a base64 JPEG plus a text note with its pixel dimensions. ALL coordinate-taking tools (mouse_move, *_click, scroll) expect coordinates in THIS image's pixel space — origin (0,0) top-left, x right, y down — so read target positions directly off the returned image; subsequent clicks are mapped through it automatically. PREFER window="" (substring of its title or app name) over the whole display whenever you are working inside one app: a full-display shot is downscaled to ~1280px wide, so small UI (list rows, sidebar items, buttons) becomes only a few pixels — too small to read or click accurately. A window capture is larger and sharper, returns a smaller image (fewer pixels → less context, faster turn), and clicks map into the window automatically. marks is ON by default: it boxes+numbers actionable elements (needs Accessibility) and lists each as [N] for compatibility with click_mark. Prefer a fresh ax_read + ax_activate for semantic controls. A magenta coordinate grid (for reading x/y) is shown automatically when marks is off and hidden when it is on; pass grid=true to force both, or marks=false for pure coordinate mode. If a target is still too small to click, use zoom_region to magnify part of this image.

Inputs: grid, marks, window.

scrollC

Scroll at the given (x, y) position. Positive lines = up, negative = down.

Inputs: x (required), y (required), lines (required), background.

type_textC

Type a string of text into the currently focused element.

Inputs: text (required), background.

waitB

Wait for a specified number of seconds before returning.

Inputs: duration.

write_clipboardC

Write text to the system clipboard.

Inputs: text (required), background.

zoom_regionA

Zoom into a rectangle of the CURRENT image (the last screenshot's pixel space) and re-capture it at native resolution — a sharp, legible magnified view. Only that rectangle is captured (not the whole display), so it is also smaller and quicker to take than a full-display shot. Use it to read exact positions on surfaces that expose no marks (canvas/games, custom-rendered views) before clicking, or to stay scoped while working inside one area. Pass x, y, width, height in the current image's pixels (width,height > 0). The returned image becomes the new coordinate space, and clicks afterward map into the zoomed region automatically. marks defaults OFF (read coordinates off the overlaid grid); grid defaults ON. Take a screenshot first so there is an image to zoom into.

Inputs: x (required), y (required), width (required), height (required), +2 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.2/5.0

    Scored across 35 tools

    Disambiguation2/5

    Two parallel semantic pipelines cover the same Chrome automation ground: ax_read/ax_activate with target=paired_page versus chrome_read/chrome_activate, with no clear rule for when to pick which. Clicking a UI element can also be done through ax_activate, chrome_activate, click_mark, or left_click, and legacy ax_click/ax_focus/ax_set_value plus the read_ui alias add further alternatives. Descriptions label preferred vs legacy vs compatibility routes, but the boundaries remain blurry enough to invite misselection.

    Naming Consistency3/5

    There is a common verb_noun pattern in tools like read_clipboard, write_clipboard, click_mark, open_application, type_text, zoom_region, and inspect_app, and the ax_ and chrome_ families are internally consistent. However, many tools break that pattern: action-only names (scroll, wait, screenshot, ocr), noun_noun names (cursor_position, batch_actions, key_combo), and adjective_noun names (left_click, right_click, double_click). The set is readable but lacks one predictable convention.

    Tool Count2/5

    At 35 tools, the server exceeds the well-scoped 3-15 range and sits in the too-many category. Several tools are explicitly legacy, compatibility, or debug variants (ax_click, ax_focus, ax_set_value, read_ui, click_mark, dump_ax), and the ax_* versus chrome_* Chrome pipelines duplicate capability rather than each earning a distinct place. The broad computer-use domain justifies a rich surface, but this set could be consolidated.

    Completeness4/5

    Coverage is strong across the apparent domain: native UI read/action, Chrome semantic lifecycle (pair, status, read, activate, focus, set value, scroll, release), visual fallbacks (screenshot, OCR, zoom), low-level input (mouse, keyboard, clipboard, scroll), and application/window discovery. Gaps such as drag-and-drop, hover, window manipulation, and conditional waits are real but minor and usually workable through existing primitives.

    Maintenance

    ActivityActive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      Enables AI agents to control macOS desktop apps via screenshots, mouse clicks, keyboard input, accessibility queries, and AppleScript.
      11
      8 npm
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Lets AI agents see and control desktop applications through the accessibility layer, enabling clicking, typing, scrolling, dragging, and window/app management across macOS, Windows, and Linux entirely on the local machine.
      25
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Lets any AI agent operate desktop apps on macOS, Windows and Linux in the background through the accessibility tree, including iPhone Mirroring on macOS. It shows the agent numbered elements plus a window screenshot, with built-in safety rules.
      3
      MIT