Nova
Nova is a single-binary MCP server that gives agents eyes-and-hands control over a real macOS or Windows desktop — screenshots, OCR, accessibility-tree reads/actions, mouse/keyboard input, clipboard, and window/app management.
Accessibility control (AX tree, background, no cursor):
ax_click(press a control by role/label substring),ax_focus(move keyboard focus),ax_set_value(set a field's value without typing), anddump_ax(debug-dump the AX tree).Screenshots & visual targeting:
screenshotcaptures the full display or a single window as base64 JPEG, with optional Set-of-Mark numbering (marks) and a coordinate grid;zoom_regionmagnifies and re-captures part of the last image at native resolution.Click-by-number:
click_markactivates an element by its mark number from the latestscreenshot(marks=true), preferring the page's own JS engine for web content and the AX tree for native controls.OCR:
ocrreads on-screen text via Apple Vision (macOS) or Windows.Media.Ocr, returning each line with a clickable center in screenshot pixel space.Mouse input (screenshot coordinate space):
left_click,right_click,double_click,mouse_move,scroll, pluscursor_positionfor OS-global logical coordinates.Keyboard input:
type_text(types into the focused element) andkey_combo(e.g. "cmd+c", "shift+tab").Clipboard:
read_clipboardandwrite_clipboardfor system text.Windows & apps:
list_windows,list_applications, andopen_applicationto launch or focus an app by name.Batching & timing:
batch_actionsruns an ordered sequence of mouse, scroll, key, type, and wait actions in one call, andwaitpauses for a given number of seconds.Many input tools accept a
backgroundflag to deliver events to the captured window's process instead of foreground.
Enables automation of the Arc browser via its web DOM bridge, allowing programmatic clicking of web elements and other interactions on web page content.
Provides the ability to automate interactions with the Safari web browser through its own JavaScript engine, enabling clicking links and buttons on web pages without needing coordinates or accessibility tree for web content.
Nova
Let any MCP agent use your real Mac or Windows apps. Nova is a single Rust binary, with no Python runtime, that gives Claude Desktop, Cursor, Codex, Claude Code and other MCP clients screenshots, native OCR, keyboard and mouse input, clipboard access, window/app control and Accessibility/UIA actions on macOS and Windows.
Releases · Part of Bodhi · Bodhi desktop app · MIT
Your real desktop, not a VM: macOS 14+ (Apple Silicon and Intel) and Windows x64/ARM64.
Reads before it clicks: Accessibility/UIA controls, Apple Vision or Windows OCR, and per-window screenshots with zoom. Clicks are given in the screenshot's pixel space and mapped back to the screen.
Types any text: full Unicode keyboard input, including Chinese, Japanese, Korean and emoji.
Accessibility-first control (v0.3.0): AX-first
ax_read/ax_activatethat reject stale snapshots, a 64-stepbatch_actionslimit with structured failure reports, the Nova.app permission owner (development preview) and an optional Chrome DevTools sidecar. Latest release: v0.3.0 — Releases.
Static image · Reproduce and inspect MCP evidence
This recording shows Nova's browser tool path (fixture page and real MCP
navigation/click calls). It was captured on Linux, so it does not show native
macOS/Windows desktop control or model reasoning. The reproduction notes explain
the --npx adapter and isolated browser.
Install
Platform | Published v0.3.0 |
macOS 14+ (Homebrew) |
|
macOS 14+ (manual) |
|
Windows x86_64 |
|
Windows ARM64 |
|
Linux | Desktop control is not supported (headless protocol checks only) |
Homebrew (macOS):
brew tap bigduu/tap
brew install bigduu/tap/nova
nova --versionThe formula installs the CLI from the release archive. It does not configure an
MCP client or grant macOS permissions. Run which nova to find the path to use
below: usually /opt/homebrew/bin/nova on Apple Silicon and
/usr/local/bin/nova on Intel Macs.
Manual download: download the matching .sha256 file from the same release
and verify the archive before extracting it. On macOS:
tar -xzf nova-v*-universal-apple-darwin.tar.gz
xattr -dr com.apple.quarantine ./nova # only if Gatekeeper blocks the download
sudo install -m 0755 nova /usr/local/bin/novaOn Windows, extract the archive for the machine's architecture and invoke
nova.exe directly or place its directory on PATH. The Windows binaries are
not Authenticode-signed, so SmartScreen may warn on first run. The macOS binary
is ad-hoc signed, not notarized.
Build from source:
git clone https://github.com/bigduu/Nova.git
cd Nova
cargo build --release --lockedThe result is target/release/nova on macOS or target/release/nova.exe on
Windows. Do not use cargo install nova: that crates.io name belongs to an
unrelated project. Nova is not published on npm.
Related MCP server: servo-mcp
Choose a version
Path | What you get |
Published v0.3.0 (Homebrew and release archives) | Screenshots, |
| Whatever lands after v0.3.0 — build from source for unreleased work. |
Nova.app remains a development preview (ad-hoc signed, not notarized). Menu-bar status and virtual cursor are labelled Preview pending live-device acceptance (#34, #70). Audit evidence.
Use it from an MCP client
These snippets use direct stdio (no arguments). They work with the v0.3.0
release and with a master build. Replace the path with your own which nova
output, the extracted binary or target/release/nova. GUI clients do not
always inherit your shell PATH, so use an absolute path.
Claude Desktop: edit
~/Library/Application Support/Claude/claude_desktop_config.json on macOS or
%APPDATA%\Claude\claude_desktop_config.json on Windows.
Cursor: edit ~/.cursor/mcp.json (all projects) or .cursor/mcp.json in
one project.
{
"mcpServers": {
"nova": { "command": "/opt/homebrew/bin/nova", "args": [] }
}
}Claude Code:
claude mcp add --scope user nova -- /opt/homebrew/bin/novaCodex: add to ~/.codex/config.toml, or run
codex mcp add nova -- /opt/homebrew/bin/nova:
[mcp_servers.nova]
command = "/opt/homebrew/bin/nova"
args = []On Windows, use the full path to nova.exe, escaped in JSON
("C:\\Tools\\nova\\nova.exe") or as a TOML literal string
('C:\Tools\nova\nova.exe').
macOS permissions: in direct stdio mode, macOS usually attributes Nova to the
app that launches it. Grant Accessibility (input and UI control) and
Screen Recording (screenshot, ocr, list_windows) to Claude Desktop,
Cursor, or the terminal/IDE that runs Claude Code or Codex. If that does not
work, add the nova binary itself. Then restart or reconnect the MCP server.
You can instead let Nova.app own the
permissions; see Managed mode.
Try: "Use Nova to list my open windows, take a screenshot of the frontmost one and tell me which buttons you can see."
Managed mode with Nova.app
nova mcp is the cross-platform managed entrypoint used by the Bamboo plugin
(v0.3.0+):
{
"mcpServers": {
"nova": { "command": "/absolute/path/to/nova", "args": ["mcp"] }
}
}Windows and Linux headless builds serve ordinary stdio MCP. On macOS, mcp
only connects to the independent Nova.app, launching it through LaunchServices
when needed. Install the app separately, open it once, and grant
Accessibility to Nova; Screen Recording is needed for capture, OCR, and
list_windows (the app menu requests it only when you explicitly choose that
action). The bundled executable can also be used as the connector:
{
"mcpServers": {
"nova": {
"command": "/Applications/Nova.app/Contents/MacOS/nova",
"args": ["mcp"]
}
}
}The explicit --connect command remains supported and uses the same transport
as macOS mcp. It carries MCP bytes over a private per-user Unix socket. The
connector does not call desktop APIs or request macOS permissions;
the app process owns the MCP handlers and TCC responsibility. The socket lives
under /tmp/nova-app-<uid>/ with a mode-0700 directory, mode-0600 socket, and a
same-UID peer check.
If Nova.app is unavailable, the managed command exits with installation and
reconnection guidance. It never falls back to desktop operations inside the MCP
host. NOVA_APP_SOCKET is for isolated development/tests; when set it disables
automatic app launch. Unset it for the normal installed-app setup. Unbundled
nova with no arguments still offers the legacy direct stdio mode.
When the app service closes its connection, the CLI connector finishes forwarding responses and exits even if the host keeps its stdin pipe open. Closing host stdin normally still half-closes the request stream and drains the service's final response, including its last buffered bytes. Standard stdout backpressure still applies: the host must keep reading responses. Reconnect only the Nova MCP server after replacing/restarting the service; Bodhi can stay open. No interrupted request is replayed and no replacement MCP session is created automatically.
This exit behavior is specific to the terminating CLI connector process.
Its dedicated runtime is released after forwarding completes, and process
exit reclaims an outstanding blocking stdin read. It does not make the
connect_stdio library function's stdin cancellable inside a resident or
embedded runtime.
Chrome DevTools MCP sidecar
For advanced Chrome page automation and debugging, Nova can launch the official
Chrome DevTools MCP
next to the desktop server. This is a transparent stdio sidecar, not a second
browser implementation inside Nova. It requires npm/npx, Node.js
^20.19.0, ^22.12.0, or >=23, and current stable Chrome (or newer). Nova
pins the reviewed upstream package to chrome-devtools-mcp@1.8.0.
Application-level integrations should accept an app selector (name or bundle ID)
and use inspect_app discovery internally. Discovery alone does not grant CDP
control; automatic app-to-provider routing remains a separate integration slice.
Endpoint copying and the separate MCP configuration below are advanced transport
compatibility. On macOS:
{
"mcpServers": {
"nova": {
"command": "/Applications/Nova.app/Contents/MacOS/nova",
"args": ["--connect"]
},
"nova-chrome-devtools": {
"command": "/Applications/Nova.app/Contents/MacOS/nova",
"args": ["chrome-devtools"]
}
}
}For a standalone binary from the v0.3.0 release or current source, use its absolute path and
["chrome-devtools"]. If a GUI client cannot find npx, add
"--npx", "/absolute/path/to/npx" after the subcommand.
The default launches a new temporary, isolated Chrome profile. Usage
statistics, package update checks, CrUX URL lookups, and sensitive network
headers are disabled/redacted by default. Requests made by attached DevTools
targets can be guarded by repeating --allowed-url-pattern, for example:
"args": [
"chrome-devtools",
"--allowed-url-pattern", "https://example.com/*",
"--allowed-url-pattern", "https://*.example.net/*"
]URL allow patterns require Chrome 149+. They apply only to DevTools targets while the MCP server is attached and are not a complete network sandbox; use an OS/VM sandbox when full network isolation is required, as described by the upstream security policy.
To work with an already running signed-in Chrome profile instead, first open
chrome://inspect/#remote-debugging in Chrome and enable remote debugging,
then configure:
"args": ["chrome-devtools", "--profile", "existing"]Automatic connection requires Chrome 144+. If several Chrome profiles are active, Chrome chooses the profile it considers the default; select and verify the connected pages before acting.
Existing-profile mode can inspect and control every open window in the selected Chrome profile, including authenticated pages. Enable it only for a trusted local MCP client, and disable remote debugging when finished.
An internal consumer or advanced caller can attach to an explicitly selected, already-running browser instead of relying on stable Chrome's default profile:
nova chrome-devtools --browser-url http://127.0.0.1:9222
nova chrome-devtools --ws-endpoint 'ws://[::1]:9222/devtools/browser/<id>'Use one endpoint, without --profile (including explicit isolated) or
--headless. --browser-url accepts HTTP(S) at the browser root, with an optional
trailing /; --ws-endpoint accepts WS(S) at /devtools/browser/<id>, where the
ID uses letters, digits, hyphen or underscore. Both require an original literal
loopback IP and explicit port 1–65535. Hostnames, abbreviated/integer/hex IPs,
credentials, query strings and fragments are rejected before starting npx.
Nova forwards the selected address literally and adds no Chrome launch flags
or automatic connection fallback. HTTPS/WSS use upstream's certificate checks.
These inputs are trusted local endpoints, not network confinement. Pinned
Puppeteer HTTP discovery
uses the returned webSocketDebuggerUrl, and its
WebSocket transport follows redirects.
A local service can therefore lead the connection to another address. The
pinned options
and ownership cleanup
distinguish attachment from launch: disconnect, stdin EOF or SIGTERM detach from
an attached browser, while upstream closes a browser it launched itself.
After connecting, call list_pages and choose by the expected title/URL. Use the
returned pageId explicitly on every page-scoped read/action. For two windows,
if the intended pages have IDs 7 and 12, first take_snapshot({pageId: 7}) and
fill({pageId: 7, uid: "<field from that snapshot>", value: "example"}); then
take_snapshot({pageId: 12}) and click({pageId: 12, uid: "<button from that snapshot>"}).
Obtain fresh page IDs and element UIDs after pages close or change; do not rely
on whichever window is focused.
The endpoint must expose browser-level CDP, covering the app's pages/windows.
The WS path check excludes renderer-only /devtools/page/... sockets and Node/V8
main-process inspector WebSocket URLs. HTTP root syntax alone cannot establish
the advertised service's identity. Upstream officially targets
Chrome/Chrome for Testing; Electron/CEF attachment is experimental and unverified
for each runtime until reads/actions are tested in its own windows. A successful
MCP tools/list alone does not establish application compatibility. On setup or
connection failure, check the selected app's supported debugging setup and
refresh its discovery metadata before retrying this transport. Keep Bodhi open;
Nova's native AX and separately paired extension capabilities remain available.
Use --enable-webmcp to expose upstream's experimental WebMCP tools. Nova adds
Chrome's required --enable-features=WebMCP launch argument in isolated mode;
for an existing profile or endpoint, the browser must already have started with that
feature enabled. WebMCP requires Chrome 150+. --expose-network-headers and
--enable-performance-crux are explicit privacy opt-ins. The pinned 1.8.0
package does not support a
--disable-javascript-evaluation option, so Nova does not advertise or pass it.
The sidecar and Nova's optional Secure Chrome Bridge serve different trust models: DevTools MCP is the broad, full-featured choice for normal browser automation, DOM/network inspection, and performance debugging; the Secure Chrome Bridge requires explicit per-page pairing and is preferable when least-privilege page scoping matters. Nova's desktop tools remain the path for browser chrome, native dialogs, and non-web UI.
Use the absolute path to the extracted release binary or the source-build
output. On Windows, use an escaped executable path such as
"C:\\absolute\\path\\nova.exe". Use the source-build output for the
AX-first workflow below. If its directory is already on PATH, the command can
be "nova".
Reconnect/reload the Nova MCP server in the client; Bodhi's main window can stay open. See Permissions & code signing for legacy direct-stdio and development-binary cases.
HTTP clients — run Nova as a server and connect over Streamable HTTP:
nova --http # 127.0.0.1:3100/mcp
nova --http --addr 127.0.0.1:8080 # custom loopback portHTTP mode is currently a local transport: it keeps rmcp's default loopback Host allowlist and does not configure remote-access authentication. Binding all interfaces is not a supported LAN setup.
First calls (current source build). Call ax_read (optionally
ax_read(window="<name>", mode="all")) for semantic content and controls, then
ax_activate(snapshot_id, node_id) on an exact actionable node. Re-run
ax_read after the action to verify semantic state. If AX/UIA coverage is
absent or partial, use focused-window ocr for rendered text; use
screenshot(window=...) / zoom_region only when pixels are necessary
(layout, icon, color, image, canvas, or visual verification). All pointer tools
use the pixel space of the most recent screenshot; cursor_position instead
reports OS-global logical coordinates.
Semantic reads (ax_read and its read_ui alias) include supported control
state and identify scroll containers with scrollable=true in all/content
mode, including unnamed containers. Containers alone receive no action mark.
The target includes its window title and, when the selected macOS window
exposes AXDocument/AXURL, an optional url. Unsupported URL/state attributes
are omitted; Windows uses supported cached UIA states and ScrollPattern metadata.
Windows reads preserve cached ValuePattern text, including Unicode and multiline values, for controls that expose the pattern. Empty, unsupported, failed, or non-string values are absent; password fields and unknown password status stay redacted. Controls exposing only TextPattern have no value fallback.
Native text writes (type_text, write_clipboard, batch typing, and
ax_set_value) omit the submitted text from tracing and acknowledgements.
Write diagnostics retain character/UTF-8 byte counts and native operation
details without echoing the value. Explicit reads such as read_clipboard
and ax_read still return the requested content.
Requirements
Linux: the source builds a headless MCP server for protocol checks; native desktop actions return an unsupported/headless error. This is not Linux desktop automation.
macOS 14+ for the macOS desktop backend. The release archive is universal and runs on Apple Silicon and Intel Macs.
Windows x86_64 or ARM64 for the Windows desktop backend. GitHub Releases provide a native archive for each architecture.
On Windows,
ocruses installed Windows OCR language packs. Usenova --ocr-langsto inspect available languages; install the needed pack if recognition reports that it is unavailable.Building on macOS requires the macOS 15 SDK / Xcode 16+ because of a transitive
apple-metalbuild dependency; that is a build-time requirement, not Nova's minimum macOS runtime version.On macOS, Screen Recording permission is required for
screenshot,ocr, andlist_windows; Accessibility is required forax_read, semantic activation, and input.
macOS grants these permissions to the process it identifies as responsible for Nova. The managed
nova mcpentrypoint and Bamboo plugin use the independent Nova.app on macOS. Direct stdio/HTTP can use the host app, terminal, or directly launched binary as the permission subject. See Permissions & code signing.
Tools
Tool | What it does |
| Canonical |
| Compatibility alias backed by the same |
| Activate an exact actionable node from a fresh |
| Capture the whole display or a single |
| Magnify a rectangle of the last screenshot at native resolution — reads small targets on surfaces with no Accessibility tree. |
| Recognize on-screen text with Apple Vision on macOS or Windows Media OCR on Windows. |
| Compatibility action for the latest numbered mark; prefer generation-safe |
| Pointer input in the pixel space of the last screenshot. |
| Read the cursor in OS-global logical coordinates; it is not converted into the last screenshot's pixel space. |
| Keyboard input (full Unicode, incl. CJK + emoji). |
| Window & app introspection. |
| Optional macOS app capability discovery. Accepts an app name/bundle ID, or discovers running Chromium candidates when omitted; no caller-supplied port or permission prompt. |
| Clipboard access. |
| Drive controls by Accessibility role/label. |
| Read the raw AX/UIA tree for diagnostics and coverage debugging. |
| Run a sequence of input actions in one call. |
| Pause for a specified number of seconds. |
batch_actions runs at most 64 actions sequentially and stops at the first
failure, without retry or rollback. Success keeps the newline-separated status
lines. Failure sets MCP isError=true and returns the same JSON in text content
and structuredContent: completed: [{index, result}], failed_index, reason,
and not_executed: {start, end_exclusive}. Indices are zero-based; the half-open
range includes only actions after the failed step, which may already have partial
side effects. More than 64 actions is rejected before any action runs, with
failed_index: null and the whole batch marked unexecuted. Each acknowledgement
or reason is limited to 512 Unicode characters, retaining its beginning and end
with … in the middle when shortened. Typed text keeps character/UTF-8 byte
counts rather than plaintext. Inspect progress and obtain fresh ax_read state
before deciding what to retry; replaying the whole batch could repeat completed
side effects.
Run
cargo run # stdio transport (default)
cargo run -- mcp # managed MCP: Nova.app on macOS, stdio elsewhere
cargo run -- --http # Streamable HTTP on 127.0.0.1:3100
cargo run -- --http --addr 127.0.0.1:8080The Swift runtime that ScreenCaptureKit links is located via an
LC_RPATHbaked in bybuild.rs, so noDYLD_*environment variable is needed forcargo run/cargo testor the standalone binary.
Nova.app development preview
Releases cut from a revision containing the app packaging workflow also attach:
nova-v<version>-universal-apple-darwin-development-app.zip
This archive contains a universal Nova.app that runs Nova's per-user app
service without a Dock icon. It gives Screen Recording and Accessibility a Nova
application identity instead of making the MCP host (for example, Bodhi) the
permission subject. Install and start it with:
shasum -a 256 -c nova-v*-universal-apple-darwin-development-app.zip.sha256
unzip nova-v*-universal-apple-darwin-development-app.zip
ditto Nova.app /Applications/Nova.app
open -gj -b com.zenith.novaInstall the app independently of Bodhi and the plugin's downloaded CLI. Keep it
at /Applications/Nova.app (or ~/Applications/Nova.app), outside Bodhi.app and
the plugin directory. The plugin still downloads the CLI archive and uses it
only as the connector on macOS; installing/updating the plugin does not install
or update Nova.app. Use a CLI and app built from the same current version.
Configure a stdio MCP client with nova mcp, as shown in
Use it from an MCP client below. If no app archive
has been published for the current code, build both macOS architectures,
combine them into a universal binary, then assemble the app with
package-development-app.sh,
which requires a universal binary and the matching Cargo version as arguments.
The app archive isDEVELOPMENT ONLY. It is ad-hoc signed, not Developer ID
signed, not notarized, and not stapled. Gatekeeper can block it, and replacing
it with a differently signed build can require granting TCC permissions again.
The existing universal CLI .tar.gz remains the supported artifact consumed
by Homebrew and Bamboo; the app .zip does not replace it.
Initial app setup and explicit local signing (0.3.1 source)
The current source adds nova app status and an explicit initial installer:
nova app status
nova app install --archive /absolute/path/to/nova-v0.3.1-universal-apple-darwin-development-app.zip \
--sha256 <64-hex-digits-from-a-trusted-release-or-catalogue>These commands are not included in the already published v0.3.0 connector. Build this source or use the matching 0.3.1 release when published; installing the existing v0.3.0 Bamboo plugin does not invoke setup or install Nova.app.
Both commands print one JSON report with state, success, message, and
next_action; valid apps also report their path, version, signing type,
development, and connector_compatible. Status checks the current account's
~/Applications/Nova.app, then /Applications/Nova.app, without creating
directories, starting the app, capturing the desktop, or requesting permissions.
States are missing, installed, already_installed, unsupported, and
error; refused or unsupported installation exits unsuccessfully.
Installation accepts a local ZIP and independently supplied SHA256. It checks
the archive, bundle identity, both Mac CPU architectures and code signature,
and requires the app version to match this connector. It publishes only an
absent ~/Applications/Nova.app, refusing any existing canonical app, symlink,
or unsafe installation directory. Source ZIP bytes stay unchanged. By default,
app metadata and download quarantine are retained, and no signing identity is
accessed. SHA256 verifies the selected bytes; obtain it from a trusted publisher.
There is no replacement or update command, automatic launch, or privacy grant.
For the explicitly selected local_ad_hoc distribution path, add
--local-sign to the same install command. After hash, ZIP, app identity,
architecture, original signature and version checks pass, setup removes only
the private staged app's download quarantine and ad-hoc signs its executable
and app, inside out, with the fixed com.zenith.nova identifier. It then verifies
the signatures before initial publication. The supported app's only executable
is Contents/MacOS/nova; unexpected nested executables are rejected.
This mode never accesses a keychain, creates a certificate, changes certificate
trust, or grants privacy permissions. Signing or quarantine errors return an
error JSON report with manual guidance and clean up the private stage. The
original archive and any existing canonical app remain unchanged.
Both the original development archive and a local_ad_hoc installation report
signing: "ad_hoc" and development: true. Neither is Developer ID signed or
Apple-notarized. Ad-hoc signatures depend on the app's bytes: the fixed bundle
identifier does not establish TCC continuity across app updates. Status stays
passive. Open Nova.app through macOS and grant the permissions you need, then
reconnect only Nova MCP; Bodhi can remain open. Existing apps, including
/Applications/Nova.app, remain untouched even with --local-sign. Production
Apple signing, upgrades and real TCC acceptance remain separate work tracked in
#36.
Permissions & code signing (macOS)
Inspect an application's interaction options
Use inspect_app when setting up an application or checking which interaction
route is available. It is optional; ordinary native interaction still starts
with ax_read.
{"app": "Slack"}The selector accepts a running application's name or bundle identifier. Exact
matches take priority over partial matches. Omit app to discover running
Electron, Chromium, and CEF candidates, including applications with no discovered
debugging connection. Names alone do not confirm a runtime: Nova checks known
framework containers and their executable evidence. Unknown or unreadable
bundles remain unknown.
The default result contains application identity, runtime, inspection status, the currently available native route, and a next step. Nova finds process-owned local connection candidates internally; callers do not need to find or supply ports. For diagnostics only, use:
{"app": "com.example.application", "details": true}Detailed output includes bundle/runtime evidence, process start identities,
endpoint provenance, and metadata verification. Nova checks the selected app's
owned listeners, recognized debugging flags, and the exact DevToolsActivePort
file only when a --user-data-dir flag evidences the profile location. It does
not scan profile contents or return full arguments/environment. Programmatically
enabled ports can be discovered through listener ownership even when a flag is
absent from the OS argument list.
browser_endpoint_available means a metadata-only browser handshake succeeded;
it does not attach browser tools, grant authorization, or verify the full
Chrome DevTools MCP toolset. Native ax_read still uses Accessibility, and the
result reports when that permission is needed. Node inspector endpoints,
incompatible endpoints, stale evidence, and incomplete inspection remain
distinct. No discovered port is not proof that debugging is disabled. Enablement
and whether a particular application can support a restart-based change remain
unknown until verified for that application.
Discovery does not launch, focus, quit, or restart applications, request
permissions, modify bundles/arguments, or open a debugging service. Network
requests stay on verified process-owned loopback sockets: /json/version,
Browser.getVersion, and Target.getBrowserContexts only. There is no page
enumeration, script evaluation, input, or Browser.close. HTTP proxies and
redirects are disabled; advertised WebSockets must keep the same owned address
and port. Ownership/start identity is checked before and after probing.
An investigation allows 8 seconds overall, 16 result apps, 32 processes per app,
4 helper generations, 8 endpoint probes per app, and 2 evidenced profiles. Each
metadata probe has a 900 ms deadline; HTTP bodies and WebSocket messages are
limited to 32 KiB, the WebSocket exchange to 128 KiB and 16 frames per reply.
Framework lookup is limited to 64 entries in an app's Contents/Frameworks,
plus at most four version directories in each recognized framework. Limits
or unavailable evidence are reported as incomplete, rather than silently
claiming that an application has no debugging support. Concurrent calls receive
a busy result. Windows/Linux return an explicit unsupported result; their
existing native tools are unchanged.
On macOS, resident desktop transports keep the process main run loop active,
so applications launched or quit after the first inspection appear or disappear
without restarting Nova or its MCP host. Relaunching an application triggers
fresh process/start-time and endpoint ownership checks. This inventory refresh
does not require Screen Recording or Accessibility permission. The mcp and
--connect byte proxies return before this desktop event loop and bootstrap.
The automated tests use fake bundles, process records, and loopback services.
The ignored own_listener_and_process_start_identity_match test inspects only
its own process/listener. The ignored e2e_app_inspection acceptance test requires
an explicitly prepared app with a dev.nova.acceptance.* bundle identifier and
NOVA_TEST_APP_BUNDLE_ID; it never defaults to inspecting the user's running
applications.
cargo test --test e2e_resident_app_inspection runs a separate macOS regression
whose test binary owns the real process main thread. In one resident process it
seeds discovery, launches a unique temporary AppKit app, checks appearance,
quits it, checks disappearance, and checks a new process identity on relaunch.
It repeats this with an internally allocated random listener that simulates
the narrow CDP handshake; the fixture is not Chromium and is reported as an
unknown runtime. It creates no windows and requests no permissions. Real
Electron/Chromium lifecycle acceptance remains a separate check. The test-only
--without-main-loop argument is a negative control that reproduces the old
stale-inventory failure; it is expected to fail.
Permission ownership
The Nova.app menu is a Preview in v0.3.0; real-desktop acceptance is tracked in #34. The packaged macOS app has a Nova menu-bar entry. It shows the local service's Starting, Ready, or Failed state separately from Accessibility and Screen Recording. Ready means the local service is listening; it does not imply either permission is granted or Chrome is paired. If startup fails, the menu stays available with a failed state. A duplicate launch exits and leaves the existing service owner running.
Startup and Refresh Status only check current permissions. They do not capture anything, open Settings, request a permission, or wait on the Chrome pairing queue. Request Accessibility… asks only for reading and operating native application controls. Request Screen Recording… asks only for screenshots and screen text recognition. Each section also has its own Open … Settings action. A “Not granted” result may mean permission has not been requested yet; it does not distinguish that case from denial.
After changing a permission, choose Refresh Status and retry the Nova tool; Bodhi can remain open. Capture continues to use the existing helper and its permission-change recovery on the next capture request; refresh itself neither starts nor restarts a capture helper. Quit Nova ends the service and its connections. Reopen Nova and reconnect only the Nova MCP server in the client when needed. There is no automatic request replay or restart control.
The menu is available only in the macOS app service. Direct stdio/HTTP and connector processes keep their existing transports and have no status menu. This UI does not guarantee that permissions survive replacing/signing Nova or an OS update; those remain separate installation and release checks.
The independent app transport is the preferred permission model: grant
Screen Recording and Accessibility to Nova.app, then use
nova mcp (or explicit nova --connect). The connector never initializes
CoreGraphics or Accessibility, so Bamboo, Claude Desktop, and terminals no
longer need Nova's desktop permissions.
Keep Nova.app installed independently and unchanged when upgrading Bodhi. The new Bodhi/plugin connector connects to the same app-owned service, so its own build/signing identity does not become Nova's permission subject. This is an architectural guarantee about where desktop calls execute; signed installation and real TCC upgrade acceptance remain separate release gates. Replacing Nova.app itself, changing its signature, or an OS permission decision can still require granting permissions again. The development preview is ad-hoc signed.
After granting Nova permissions in System Settings, retry the tool. If macOS requires a restart for the change, quit/reopen Nova.app, then reconnect only the Nova MCP server in the client. Keep Bodhi's main window open. The connector does not replay interrupted requests or automatically restore an MCP session after Nova exits. Do not remove/re-add Bodhi's grants to repair this managed Nova path.
Two details still matter for direct stdio/HTTP and source-development modes:
Grant the responsible process for the way Nova is launched. macOS TCC may
attribute a child process to its responsible parent app. For legacy direct
stdio MCP (an empty argument list), grant Claude Desktop, Bamboo, or the
terminal/IDE that launches Nova. For a directly launched CLI/HTTP process, macOS may instead use the Nova
binary. If granting the expected host does not work, add the installed nova
binary (or target/release/nova) as a fallback under System Settings → Privacy
& Security → Screen Recording and Accessibility.
Keep the identity of whichever process receives the grant stable. If Nova
itself is the permission subject, cargo build produces an ad-hoc,
linker-signed binary whose code-signing identity is a content hash
(nova-<hash>). It changes every build, so a direct binary grant stops applying.
Sign Nova with a stable self-signed identity when developing in that mode:
cargo build --release
./scripts/dev-codesign.sh --release # re-sign after EVERY buildThe first run creates a Zenith Nova Code Signing identity in your login keychain
(click Always Allow once if codesign prompts) and signs the binary with a
fixed identifier (com.zenith.nova). A direct Nova grant then survives rebuilds
that are re-signed with the same certificate. Host-app grants likewise depend on
the host keeping a stable signing identity.
Troubleshooting —
screenshotfails with a "wedged" / "busy" capture error. All captures (and window enumeration) run in ONE shared per-user daemon (nova --capture-daemon, flock-elected, socket/tmp/nova-capture-<uid>-<hash>.sock), becausereplaydkeys clients by executable path — two same-binary ScreenCaptureKit clients evict each other's XPC identity and wedge every new stream start. The daemon kills itself if a capture exceeds its 8s watchdog, and the client auto-recovers: kill+respawn the daemon, then (second failure) SIGKILL all nova capture processes andkillall -9 replayd— wedges self-heal without manual action. If they don't:nova --selftest(probes ScreenCaptureKit in a sacrificial subprocess, then the daemon path) and read/tmp/nova-capture-worker.log(step trace) +/tmp/nova-capture-daemon.log(daemon stderr). Manual remedy = kill the processes holding streams (pkill -f -- --capture-daemon), NOT replayd: plainkillall replaydis a no-op (replayd ignores SIGTERM), and evenkillall -9 replayddoesn't cure a wedge while a stream-holding client survives — it just reconnects and re-wedges the fresh replayd.
Coordinate grounding
The virtual cursor below is part of the Nova.app Preview in v0.3.0; real-desktop acceptance is tracked in #70. In macOS Nova.app, coordinate-backed mouse movement, clicks and scrolls also show a purple virtual arrow at the supplied logical point. Click rings and scroll direction cues fade after 400 ms; the arrow clears after 1.2 seconds of inactivity. The panel passes through mouse input and cannot take keyboard focus. Foreground input still moves the real pointer; PID/background delivery keeps its existing behavior. Batch uses the same native backend. Internal pointer restoration after element-center fallback leaves the cue at that click. Feedback marks an input attempt; it does not confirm the target accepted it. Semantic-only AX actions and browser DOM actions do not infer cursor positions; direct no-UI transports and pure connectors do not create this overlay.
Display and region captures exclude the reserved cursor window only when its owner resolves to the same Nova executable, including updates to a warm capture stream. Other-app single-window captures keep their existing filter. Rendering, click-through, focus/pointer behavior, batch, cold/warm screenshot exclusion and Quit cleanup still require controlled real desktop acceptance; automated tests do not establish those GUI results. Multi-display and full-screen behavior also need live verification on the available setup.
A general LLM judging pixel coordinates off a downscaled screenshot is the main source of mis-clicks — so the primary path avoids pixels entirely.
ax_readfirst (no image) — returns actionable controls and non-actionable readable content in deterministic tree order. A successful macOS read requires Accessibility but does not contact ScreenCaptureKit.permission_deniedmeans fix that grant; it is not an instruction to take a screenshot.Fresh semantic action — call
ax_activatewith the returned snapshot and node IDs. Native AX/UIA and the browser DOM bridge are tried before a freshly revalidated element-center click. Stale generations fail closed; every activation attempt consumes its generation before provider dispatch, so read again after any result.OCR second — when coverage is absent/partial and the missing information is rendered text, use focused-window OCR and its returned text center with
left_click(..., source="ocr_center").Screenshot/zoom last — use pixels for visual-only state or a surface with no semantic/text representation; coordinate clicks report
route=visual_coordinate. Screenshot marks andclick_markremain available for compatibility.
When a screenshot is needed, all click/move/scroll tools work in the pixel space of the last screenshot — the server remembers that frame and maps clicks back to the real screen, so the model just "clicks what it sees":
screenshot(window: "<name>")— capture a single window (substring of its title or app name) instead of the whole display. Smaller, sharper image → less context and far less downscaling → better precision. Later clicks map into that window.zoom_region(x, y, w, h)— magnify a rectangle of the last screenshot at native resolution (capturing only that rectangle). For reading small targets on surfaces that expose no Accessibility tree (canvas, games, custom views), where coordinates are the only option. A labeled coordinate grid is overlaid so the model reads positions straight off the axes.
Testing
The suite is split into fast, hermetic tests (run by default) and side-effecting
end-to-end tests (opt-in, #[ignore]d).
Default — unit + hermetic integration tests
cargo testRuns everything that has no side effects and needs no special permission:
unit tests for coordinate scaling, the key/char keystroke maps, combo parsing, batch (de)serialization, and MCP tool registration;
tests/e2e_interaction.rs— screenshot→logical coordinate mapping (viaCGDisplay, no permission needed) and a non-destructive clipboard round-trip (snapshots and restores the clipboard).
This is what the macOS test job runs in CI (see .github/workflows/ci.yml;
the workflow also has Windows cross-check and Linux headless jobs).
End-to-end tests (#[ignore]d)
These either post real input events (they move the cursor, click, scroll, or
type into the focused window) or require Screen Recording permission, so they
are excluded from cargo test and must be opted into. Run them on a desktop
session where that's acceptable:
# all of them
cargo test -- --include-ignored
# or a single one
cargo test --test e2e_input mouse_move_roundtrips_through_cursor_position -- --ignoredTest (file) | What it does | Needs |
| Resolves and reads a focused or | Accessibility / logged-in UIA desktop |
| Moves the cursor, reads it back via | Accessibility |
| Left/right/double click on the empty desktop corner (Esc dismisses the menu) | Accessibility |
| Posts vertical scroll events | Accessibility |
| Types into the focused window | Accessibility |
| Launches/focuses System Settings | — |
| Enumerates on-screen windows | Screen Recording |
| Captures the display, checks the JPEG | Screen Recording |
| Asserts capture dims match the click-coordinate mapping | Screen Recording |
| Captures a window and validates its view-frame metadata | Screen Recording |
| Runs Apple Vision OCR on a live capture; asserts text + in-bounds line centers | Screen Recording |
| Shared capture daemon: capture, kill→respawn recovery, concurrent clients, clean-error survival | Screen Recording |
| Old | Screen Recording |
| Exercises the stdio (JSON-RPC) transport end-to-end | — |
| Launches Safari, opens Google, and reads the page through Nova | Network + Screen Recording + Accessibility |
mouse_move_roundtrips…proves the macOS pointer post and cursor read-back round-trip using logical coordinates. The non-ignored interaction test covers screenshot→logical coordinate arithmetic, while the live screenshot tests cover captured-dimension contracts.Run
e2e_capture_workersingle-threaded (-- --ignored --test-threads=1): the tests share one daemon/socket.
list_applications_returns_app_bundles (in e2e_input) is not ignored — it
only reads Spotlight and is tolerant of a Spotlight-less CI host.
Lint & format
cargo fmt --all -- --check
cargo clippy --all-targetsReleasing (maintainers)
A version tag drives everything via .github/workflows/release.yml. The
workflow resolves the tag once, verifies it against Cargo.toml and the event
commit, and makes every source-building job check out that immutable commit. It
builds and smoke-tests the universal macOS CLI and development-only Nova.app,
creates the Release with those assets, then sequenced jobs attach native Windows
x86_64/ARM64 archives and the Bamboo plugin bundle. The CLI .tar.gz name and
checksum outputs stay unchanged for Homebrew and the Bamboo plugin
manifest.
Run the hermetic release checks before tagging:
scripts/test-release-workflow.shThe published release is v0.3.0. When cutting the next release, make sure the tag matches the manifest version. Release tags must be protected from force updates; the workflow also serializes runs by tag and re-verifies the tag before its first upload. The Nova.app asset must remain labeled DEVELOPMENT ONLY until all production distribution gates are complete:
sign nested code and the outer app, in that order, with a Developer ID Application identity and the hardened runtime;
submit the distribution artifact to Apple's notary service and verify the accepted ticket;
staple the ticket to the app and validate it with
codesignandspctl;authenticate local MCP and Chrome bridge peers with macOS audit tokens and designated code requirements, rather than relying on same-UID sockets alone;
run the packaged native host and extension against a real Chrome install, including pairing, navigation revocation, stale snapshots, and disconnects;
pin third-party GitHub Actions by full commit SHA before treating the release workflow as a production supply-chain boundary;
smoke-test launch, upgrade,
nova --connect, Screen Recording, and Accessibility grants on clean Apple Silicon and Intel macOS 14+ machines.
Do not describe the ad-hoc-signed app preview as a production-ready macOS app.
License
MIT © bigduu
Available Tools
35 toolsax_activateA
Activate one actionable node from the immediately preceding ax_read using its snapshot_id and node_id. This generation-safe action fails closed when the UI has been read again. Paired-page nodes use their captured Chrome Session/route and provider IDs; DOM errors never use native/coordinate fallback. Native nodes use the web DOM bridge for scriptable browser content, native AX/UIA activation otherwise, then a freshly verified element-center fallback. The result reports route=chrome_extension|ax|uia|web_dom|element_center. An accepted current actionable token consumes that generation before provider dispatch, so rerun ax_read after any result.
| Name | Required | Description | Default |
|---|---|---|---|
| node_id | Yes | Snapshot-local node id (for example "n7"). The node must be actionable. | |
| background | No | Applies only to the verified element-center fallback. Semantic AX/UIA/DOM activation is already background and cursor-free. | |
| snapshot_id | Yes | Ephemeral snapshot id returned by the immediately preceding ax_read. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so: it discloses generation-safe fail-closed behavior, the provider-specific routing rules (paired-page vs native), that DOM errors never use native/coordinate fallback, the fallback chain, and the route values returned. This is unusually rich behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Dense but front-loaded: the core action and its scoping constraint lead, followed by safety and routing details. Every sentence carries information, though the density is high and slightly demanding, keeping it just below ideal concision.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no output schema, the description explains the result format (route=chrome_extension|ax|uia|web_dom|element_center) and the post-call requirement to rerun ax_read. For a tool this complex, the agent has everything needed to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents all three parameters including the background fallback-only note. The description reinforces that snapshot_id is ephemeral/immediately-preceding and node_id must be actionable, but adds little beyond the schema's own descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Activate one actionable node from the immediately preceding ax_read') and scopes it with the required snapshot_id and node_id. It is clearly distinguishable from siblings like ax_click and chrome_activate, which do not have the generation-safety constraint.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear context: activation must follow an immediately preceding ax_read, and the agent must 'rerun ax_read after any result' because an accepted token consumes the generation. It does not explicitly contrast with ax_click or ax_focus (click vs activate), so it falls short of full when/when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ax_clickA
Legacy query action through macOS Accessibility or Windows UI Automation — no coordinates, no cursor movement, works in the background (the app need not be frontmost). query is a case-insensitive substring of the element's accessibility role or label/title (e.g. "Send", "Search"). Ambiguous matches fail closed. Prefer fresh ax_read + ax_activate for exact, generation-safe actions.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | Case-insensitive substring matching the target element's accessibility role or label/title (e.g. "Send", "Search", "AXButton"). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and does well: no coordinates, no cursor movement, app need not be frontmost, and ambiguous matches fail closed. It does not mention required accessibility permissions or what a successful call returns/reports, which are meaningful remaining gaps for a UI-automation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with the mechanism and the key routing warning; each sentence carries distinct information (mechanism, matching semantics, failure mode, alternative). Dense with backticked identifiers but no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter, no-output-schema tool with no annotations, the description covers mechanism, matching semantics, failure behavior, and sibling routing. The main omission is any hint of the return value or post-action confirmation, minor given the absence of an output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single parameter is already fully documented in the schema. The description restates the matching rule and adds the useful 'ambiguous matches fail closed' constraint, but contributes little parameter detail beyond the schema, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific action (click/query against an element) and the underlying mechanism (macOS Accessibility / Windows UI Automation), and explicitly names the preferred alternatives (ax_read + ax_activate). The phrase 'legacy query action' is slightly loose about what the action does (a click), but the surrounding sentences make the semantics clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear negative directive — prefer fresh ax_read + ax_activate for exact, generation-safe actions — which frames this as the legacy fallback path. It also notes when it is appropriate (background operation, no coordinates/cursor). No explicit statement of when the legacy path is the *only* option, but the routing intent is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ax_focusA
Legacy query action that focuses a control through macOS Accessibility or Windows UI Automation. Background, no cursor. Ambiguous substring matches fail closed; prefer a fresh ax_read node whenever possible.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | Case-insensitive substring matching the target element's accessibility role or label/title (e.g. "Send", "Search", "AXButton"). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden, and it does disclose non-obvious traits: background execution, no cursor movement, and fail-closed behavior on ambiguous matches. It omits permission/accessibility-authorization requirements and what a successful call returns.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight clauses, front-loaded with what the tool does, then side-effect profile, then the alternative recommendation. No filler, though 'Legacy' is somewhat imprecise about what it supersedes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter tool with no output schema and no annotations, the description covers action, side effects, and failure mode adequately but leaves the return value and authorization/permission needs unspecified — the latter being genuinely relevant for OS accessibility APIs.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the single 'query' parameter is already fully documented as a case-insensitive substring. The description's 'ambiguous substring matches fail closed' adds a small amount of resolution semantics beyond the schema, but the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource (focuses a control) plus the platform mechanism (macOS Accessibility / Windows UI Automation). The word 'Legacy' and the pointer to ax_read differentiate it from sibling focus/activate tools, though it doesn't explicitly compare against ax_activate or ax_click.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear routing rule — 'prefer a fresh ax_read node whenever possible' — which steers the agent toward an alternative, and 'Legacy' implies a fallback status. It stops short of stating explicit preconditions (e.g., when focusing is actually required vs. activating).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ax_readA
Canonical ax:read capability. target=paired_page selects only the already consented Chrome document, before native targeting; mutually exclusive with window. Page nodes have provider=chrome_extension and no native marks/global bounds; page failures require inspection/re-pairing and never native or pixel fallback. target=native (default) reads visible semantic UI directly from macOS Accessibility or Windows UI Automation without taking a screenshot. Returns an ephemeral snapshot_id, deterministic node ids, role/name/description/value/actions/state, optional global-logical bounds, explicit coverage/fallback status, and actionable marks. mode=all (default) combines controls with labels/static text/headings; interactive or content narrows the view. Call ax_activate(snapshot_id,node_id) on a fresh actionable node. If coverage is absent/partial, use focused OCR for missing rendered text, then screenshot/zoom only for visual-only pixels. permission_denied means grant Accessibility; do not hide it with a screenshot fallback.
| Name | Required | Description | Default |
|---|---|---|---|
| max | No | Cap on the number of elements returned (default 200, max 400). | |
| mode | No | `interactive` returns actionable controls, `content` returns readable labels/text, and `all` combines both in deterministic tree order. Defaults to `all`. | |
| filter | No | Show only elements whose role, label, or value contains this case-insensitive substring (e.g. "button", "search", "submit"). Omit to list everything. Filtering only narrows the DISPLAY — the hidden elements keep their numbers and stay clickable by click_mark(N). | |
| target | No | `native` (default) reads AX/UIA. `paired_page` reads only the already consented Chrome document; mutually exclusive with `window`. | |
| window | No | Target a single on-screen window: a case-insensitive substring of its title or app name (e.g. "Safari", "Settings"). Omit to read the current target app — the last window-captured or read_ui'd app, otherwise the frontmost. Web-page content is included either way (as long as the app has a window); a window= just narrows and sharpens the listing to that window. | |
| max_chars | No | Unicode-character output budget (default 30000, max 100000). If hit, `truncated=true` and `partial_reason=character_limit` are returned. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and delivers: no screenshot is taken, page (paired_page) failures never fall back to native or pixel paths, coverage/fallback status is reported explicitly, and permission_denied requires granting Accessibility. It also discloses the ephemeral snapshot_id and the deterministic node-id model that downstream ax_activate depends on.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
It is information-dense with little waste, but it is a long unparagraphed block and the opening sentence ('Canonical ax:read capability') is low-value. Several clauses duplicate schema-level detail rather than adding new guidance, so structure and conciseness are only adequate.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description must describe returns, and it does: ephemeral snapshot_id, deterministic node ids, role/name/description/value/actions/state, optional global-logical bounds, coverage/fallback status, and actionable marks. For a 6-parameter, zero-required, read tool this is complete enough to call correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so each parameter is already documented. The description reinforces key semantics (target=native default, paired_page mutually exclusive with window, mode=all default) but mostly restates what the schema already provides. Baseline 3 is appropriate when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource (reads visible semantic UI via macOS Accessibility / Windows UI Automation) and distinguishes itself from screenshot-based and Chrome-document reading paths (target=paired_page vs native). The opener 'Canonical ax:read capability' restates the name, but the rest pins down exactly what is read and how. It does not explicitly differentiate from close siblings like read_ui or dump_ax, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear operational guidance: call ax_activate(snapshot_id,node_id) on a fresh actionable node, use focused OCR when coverage is absent/partial, then screenshot/zoom only for visual-only pixels, and treat permission_denied as a grant-Accessibility signal rather than masking it with a screenshot. These are strong when-to-use and sequencing rules. It omits explicit routing against sibling readers (read_ui, dump_ax, chrome_read), which keeps it from a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ax_set_valueA
Legacy query action that sets a control value through macOS Accessibility or Windows UI Automation. Background, no cursor. Ambiguous substring matches fail closed; prefer a fresh ax_read node whenever possible.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | Case-insensitive substring matching the target element's role or label. | |
| value | Yes | The value to set (e.g. the text to place into a field). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and delivers useful traits: background execution, no cursor movement, legacy status, and fail-closed behavior on ambiguous substring matches. It omits permission requirements and what a successful/partial set reports.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three dense sentences with the operation, its safety behavior, and the alternative all front-loaded; no filler. The closing advisory is slightly terse but earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a no-annotation, no-output-schema mutation tool, the description covers mechanism, side-effect profile, and failure semantics well. It could still say more about permissions and post-condition reporting, but nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds real meaning beyond the schema by explaining that ambiguous substring matches against `query` fail closed rather than picking arbitrarily.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (sets) and resource (a control value) plus the underlying mechanism (macOS Accessibility / Windows UI Automation). 'Legacy query action' and the reference to ax_read make it distinguishable from the many sibling click/read/type tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly steers the agent away from this tool in favor of 'a fresh ax_read node whenever possible,' signaling a deprecation/preference order. It stops short of spelling out the conditions under which this legacy path is still required.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
batch_actionsA
Execute a sequence of input actions (mouse_move, left_click, right_click, double_click, scroll, key_combo, type_text, wait) in one call to reduce round-trips, at most 64 actions. Coordinates are in screenshot space. Stops at the first failure and returns completed indices/results, failed_index/reason, and a not_executed half-open range. The failed action may have partial side effects. No retry or rollback. Get fresh ax_read state before deciding what to retry; use a screenshot for visual outcomes.
| Name | Required | Description | Default |
|---|---|---|---|
| actions | Yes | Up to 64 ordered input actions. Larger batches are rejected before any action runs. Execution stops at the first failure. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears the full behavioral burden. It discloses critical traits: at most 64 actions, stop-at-first-failure semantics, returned fields (completed indices/results, failed_index/reason, not_executed range), possible partial side effects, and no retry or rollback. This is unusually rich and directly useful for safe invocation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose and constraints, then proceeds to failure behavior and retry guidance. It is dense but every sentence adds information; no obvious padding. A minor reordering or break could improve scanability, but it is efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex batch tool with no output schema and no annotations, the description is remarkably complete. It explains what happens on failure, what is returned, and how to verify results before retrying. An agent has enough context to call and recover from this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the schema already documents the actions array, its maxItems limit, and the BatchAction variants. The description repeats the max of 64 and the action list, and adds that coordinates are in screenshot space, but this largely duplicates structured data. Baseline 3 is appropriate when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Execute a sequence of input actions'), lists the exact action types, and gives the core purpose ('in one call to reduce round-trips, at most 64 actions'). This clearly distinguishes batch_actions from the individual action siblings like left_click, mouse_move, and type_text that it aggregates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides clear context for use ('in one call to reduce round-trips') and post-failure guidance ('Get fresh ax_read state before deciding what to retry; use a screenshot for visual outcomes'). It does not explicitly state when not to use it (e.g., for a single action) or compare directly to the individual tools, so it stops short of full when/when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chrome_activateA
Semantically activate one exact actionable node from the immediately preceding chrome_read snapshot. Fails closed on stale snapshot, route, capability, or DOM identity; coordinate fallback is forbidden.
| Name | Required | Description | Default |
|---|---|---|---|
| node_id | Yes | Exact snapshot-local semantic node id. | |
| snapshot_id | Yes | Ephemeral snapshot returned by the immediately preceding chrome_read. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so well: it declares fail-closed behavior on stale snapshot, route, capability, or DOM identity, and forbids coordinate fallback. It stops short of describing what activation actually performs (click/keypress) or any return/confirmation signal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense sentences with zero waste; the core action is front-loaded and the failure/prohibition constraints follow immediately.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter mutation tool with no annotations and no output schema, the definition covers preconditions, failure modes, and the prohibited alternative. The remaining gap is what a successful activation returns or how success is confirmed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so snapshot_id's ephemeral/perishable nature and node_id's snapshot-local exactness are already documented in the schema. The description reinforces these constraints but adds no syntax or format detail beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (activate) and a precisely scoped resource: one exact actionable node from the immediately preceding chrome_read snapshot. The chrome_read lineage separates it from the ax_* family, though no sibling is explicitly named as an alternative.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear precondition ('immediately preceding chrome_read snapshot') and an explicit exclusion ('coordinate fallback is forbidden'), which routes the agent away from coordinate-based tools like left_click. It does not directly compare against ax_activate, but the context is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chrome_focusA
Focus one exact focus-capable node from the immediately preceding chrome_read snapshot. Fails closed instead of guessing or clicking.
| Name | Required | Description | Default |
|---|---|---|---|
| node_id | Yes | Exact snapshot-local semantic node id. | |
| snapshot_id | Yes | Ephemeral snapshot returned by the immediately preceding chrome_read. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. 'Fails closed instead of guessing or clicking' is a genuinely useful disclosure that there is no implicit fallback and no synthetic click, but it says nothing about the visible effect of focusing (keyboard focus movement), permission requirements, or the failure signal returned.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, front-loaded with the action and target, with the failure policy second. 'Guessing or clicking' is slightly redundant with 'fails closed', keeping it just under maximum efficiency.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-required-parameter, fully documented schema with no output schema, the description supplies what the structured fields don't: the prior-call dependency and the fail-closed policy. Only the observable effect of a successful focus is left implicit.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so both node_id and snapshot_id are already documented in the schema. The description restates the snapshot lineage ('immediately preceding chrome_read') and the node constraint ('focus-capable') but adds no format or syntax detail beyond the schema, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (focus) and resource (one exact focus-capable node) and scopes it to a node from the immediately preceding chrome_read snapshot, which separates it from the AX-family sibling ax_focus. It does not name an alternative explicitly, but the chrome_read lineage plus the 'focus-capable' qualifier make the target unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a concrete precondition: the node must come from the immediately preceding chrome_read snapshot, implying chrome_read is a required prior call. No exclusions or named alternatives (e.g. when to prefer ax_focus) are provided, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chrome_pairA
Begin an explicit 30-second pairing request for the active Chrome page. The user must inspect the origin and click Pair in the Nova extension popup. Success binds the exact tab, documentId, page nonce, and new epoch; it never grants access to another tab or later navigation.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and delivers: a 30-second expiry window, the required human-in-the-loop confirmation step, the exact binding scope (tab, documentId, page nonce, new epoch), and an explicit negative guarantee that it never grants access to another tab or later navigation. That is unusually rich behavioral disclosure for a session-establishing tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the action and its time bound, followed by the confirmation requirement and binding guarantees. Every clause adds a fact an agent needs; there is no padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, and the description explains what success means, but it does not say how to detect failure or timeout, or what the agent should call next (e.g., chrome_status) while waiting for the user to click Pair. Those gaps keep it from a 5.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters and the schema is trivially complete, so baseline 4 applies. Nothing in the description is needed to clarify inputs.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource: 'Begin an explicit 30-second pairing request for the active Chrome page.' This is clearly distinct from siblings like chrome_activate or chrome_read, though the description never names an alternative to route against, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear context and a prerequisite workflow: the request only succeeds after the user inspects the origin and clicks Pair in the Nova extension popup. It does not state when not to use it or which sibling to use instead (e.g., chrome_status to poll for completion), so no exclusions are covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chrome_readA
Read a bounded semantic DOM snapshot from the exact paired Chrome document. Sensitive fields and invalid/presentation ARIA nodes are excluded. Returns an ephemeral snapshotId and exact nodeIds; run this immediately before every Chrome mutation. No screenshot or coordinates are used.
| Name | Required | Description | Default |
|---|---|---|---|
| max_chars | No | Maximum Unicode-character budget (default 100000, hard capped at 500000). The result reports truncation rather than silently omitting it. | |
| max_nodes | No | Maximum semantic nodes returned by the extension (default 500, hard capped by the content script at 1000). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so well: it discloses that output is bounded, that sensitive fields and invalid/presentation ARIA nodes are filtered, that the snapshotId is ephemeral, and that exact nodeIds are returned for use by mutations. It omits auth/permission requirements and any rate or freshness caveats, keeping it short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three front-loaded sentences that move cleanly from what it reads, to exclusions, to return value plus usage. Tight and well-ordered, with only mild redundancy in the closing screenshot/coordinates disclaimer.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, but the description compensates by naming the returned values (ephemeral snapshotId and nodeIds) and the filtering behavior. Combined with 100% schema coverage on both parameters, an agent has nearly everything needed, though permission/error behavior remains unstated.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so max_chars and max_nodes (including defaults and hard caps) are already fully documented in the schema. The description only echoes the notion of boundedness without adding syntax or interaction detail, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Read) and resource (bounded semantic DOM snapshot from the exact paired Chrome document), and the 'No screenshot or coordinates are used' clause distinguishes it from screenshot/ocr/click_mark siblings. An agent can tell it apart from ax_read and the visual tools without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit when-to-use directive: 'run this immediately before every Chrome mutation', which is strong operational guidance. It does not, however, name sibling alternatives or state exclusion conditions (e.g. when to prefer ax_read or chrome_status), so it stops short of a full when/when-not treatment.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chrome_releaseB
Explicitly release the currently paired Chrome document, revoke its epoch, in-flight requests, and unexpired receipts.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It does disclose real consequences by naming what gets revoked (epoch, in-flight requests, receipts), which is useful side-effect information. But it omits error behavior if nothing is paired, idempotency, and the resulting state, leaving notable gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence that begins with the action verb and the target, with no padding. It is slightly dense with undefined domain jargon, but every clause contributes to describing the operation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter tool with no output schema, the description covers what the operation does and what it revokes. It stops short of covering preconditions (is a pairing required?), failure cases, or whether the call is idempotent, which for an unannotated tool leaves meaningful context missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the baseline of 4 applies. The description correctly implies no arguments are needed by operating only on the 'currently paired' document, so there is no parameter semantics gap to fill.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (release) and a clearly bounded resource (the currently paired Chrome document), and the term 'paired' implicitly distinguishes it from the sibling chrome_pair. However, the jargon 'epoch,' 'in-flight requests,' and 'unexpired receipts' is not explained, which slightly muddies what is actually being released.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The word 'Explicitly' hints that this is a manual counterpart to some implicit release, but the description never states when an agent should call this versus leaving pairing to lapse. It does not name chrome_pair or any sibling as the alternative action, so usage is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chrome_scrollB
Semantically scroll one exact scroll-capable node (including root when exposed) from the immediately preceding chrome_read snapshot. Direction is up/down/left/right and amount is line/half_page/page; no coordinates.
| Name | Required | Description | Default |
|---|---|---|---|
| amount | Yes | ||
| node_id | Yes | Exact snapshot-local semantic node id, or `root` when exposed by the document snapshot. | |
| direction | Yes | ||
| snapshot_id | Yes | Ephemeral snapshot returned by the immediately preceding chrome_read. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It usefully discloses that the snapshot_id is ephemeral and must come from the immediately preceding chrome_read, and that it targets a single node, but it says nothing about side effects, what happens on a stale snapshot, or the return value.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with the core action front-loaded and parameter semantics appended; no filler. Slightly dense but every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema or annotations exist, so the description should do more. It covers the snapshot dependency and node targeting adequately but omits return behavior and failure modes, leaving an agent with gaps for a 4-required-param mutation-like tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%, with node_id and snapshot_id already documented in the schema. The description restates the direction enum (up/down/left/right) and amount enum (line/half_page/page) in prose, adding little beyond the self-descriptive enums, so it neither compensates for nor worsens the coverage gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (scroll) and resource (one exact scroll-capable node, including root), and the clause 'no coordinates' implicitly separates it from a coordinate-based scroll sibling. An agent can tell what it does, though it never names the alternative tool explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The prerequisite 'from the immediately preceding chrome_read snapshot' implies when to use it, and 'no coordinates' implies when to prefer a semantic scroll. However, there is no explicit when/when-not or named alternative, so usage is only implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chrome_set_valueA
Set one exact non-sensitive text control from the immediately preceding chrome_read snapshot. Password, payment, OTP, file, hidden, and explicitly sensitive controls are unavailable. Plaintext is never logged or echoed; success returns only UTF-8 byte length and SHA-256.
| Name | Required | Description | Default |
|---|---|---|---|
| value | Yes | New field value. Nova never logs or echoes this plaintext; the result contains only its UTF-8 length and SHA-256 digest. | |
| node_id | Yes | Exact snapshot-local semantic node id for a non-sensitive text control. | |
| snapshot_id | Yes | Ephemeral snapshot returned by the immediately preceding chrome_read. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses meaningful traits: plaintext is never logged or echoed, and success returns only UTF-8 byte length plus SHA-256. It does not state failure modes, permission requirements, or whether the write is reversible.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three dense sentences, front-loaded with the action and constraint, with no filler. Slightly compressed but each sentence carries information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutation tool with no annotations and no output schema, the description covers the snapshot freshness requirement, control-type exclusions, and the exact return shape (byte length + SHA-256), which is what an agent needs to call and interpret it. Minor gaps remain around error behavior.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents snapshot_id, node_id, and value. The description reinforces the security property of the value parameter but adds no syntax or format detail beyond the schema, which is the expected baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Set one exact non-sensitive text control') and scopes the source of the node to a preceding chrome_read snapshot, which distinguishes it from generic siblings like type_text or ax_set_value.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states the precondition (value must come from the immediately preceding chrome_read snapshot) and the exclusions (password, payment, OTP, file, hidden, sensitive controls unavailable). It does not name a fallback alternative such as ax_set_value or type_text, so routing knowledge is incomplete.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chrome_statusA
Report whether Nova's Chrome Native Messaging host is connected and whether one exact top-level document is currently paired. Call this before Chrome semantic work; it never inspects page content.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden and discloses that the call is a non-invasive status check ('never inspects page content') and what two states it reports. However, it says nothing about whether the check blocks/waits, its latency, or side effects, leaving meaningful behavioral gaps for a no-annotation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with zero filler. The core purpose is front-loaded and the behavioral caveat follows immediately, making it easy to scan and act on.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a parameterless status tool with no output schema, the description adequately states what is checked. It could go slightly further by indicating what an agent should do on a negative result (e.g., pairing via chrome_pair), which is the only notable omission.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the schema baseline of 4 applies and there is nothing further for the description to clarify. No parameter semantics are required or missing.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Report') and precisely names the resource: whether the Chrome Native Messaging host is connected and whether a top-level document is paired. This clearly distinguishes it from content-reading siblings like chrome_read and control siblings like chrome_focus or chrome_activate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit trigger ('Call this before Chrome semantic work') and a distinguishing negative scope ('it never inspects page content'), which tells the agent when not to rely on it for content. It stops short of naming the specific alternative (e.g., chrome_pair) for remediating a disconnected state, so not a full 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
click_markA
Compatibility activation by the mark NUMBER shown by the most recent ax_read/read_ui or screenshot(marks=true). Prefer ax_activate with snapshot_id/node_id because it rejects stale generations. Web-page content in a scriptable browser (Safari, Chrome, Arc, Edge, Brave, …) is clicked through the page's OWN JavaScript engine (an Accessibility press is a silent no-op on web content), native controls through the Accessibility tree; if neither applies it falls back to a click at the element's center. Numbers go stale when the UI changes, so run a fresh ax_read immediately before acting.
| Name | Required | Description | Default |
|---|---|---|---|
| number | Yes | The mark number [N] of the element to activate, as listed by the most recent `read_ui` or `screenshot(marks=true)`. | |
| background | No | Deliver the coordinate-click fallback in the background to the captured window's process (native apps only). The AX action is always background; this only affects the fallback. Default false = foreground. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so well: it explains that web content is clicked through the page's own JS engine, that an Accessibility press is a silent no-op on web content, native controls go through the AX tree, and otherwise it falls back to a center-coordinate click. It also warns that mark numbers go stale when the UI changes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three dense sentences that are front-loaded with the core purpose and the preferred alternative. Every clause conveys operational information, though the mid-sentence branch description is slightly heavy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 2-parameter tool with no annotations and no output schema, the description covers activation mechanics, fallback behavior, staleness risk, and the recommended alternative. Nothing needed to invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds lifecycle semantics the schema lacks: mark numbers are ephemeral and go stale on UI change, requiring a fresh read. The background-only fallback nuance is left to the schema, which documents it adequately.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (activation/click) plus the exact resource it consumes (the mark NUMBER from the most recent ax_read/read_ui or screenshot(marks=true)). It explicitly differentiates itself from the preferred sibling ax_activate, so an agent can route correctly without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use ('Prefer ax_activate with snapshot_id/node_id because it rejects stale generations') and a concrete precondition ('run a fresh ax_read immediately before acting'). The alternative and the reason to choose it are both named.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cursor_positionA
Get the current mouse cursor position. Returns (x, y) in logical coordinates.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose behavioral traits such as whether the operation is read-only, has side effects, or requires permissions. It only states it returns coordinates.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Very concise single sentence. Could be improved by elaborating on 'logical coordinates' or units, but currently efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple, zero-parameter tool, the description adequately states purpose and return value. However, it does not explain 'logical coordinates' or any limitations, leaving minor gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist, so schema coverage is 100%. The description adds return format information ('Returns (x, y) in logical coordinates'), which is appropriate for a zero-parameter tool. Baseline of 4 is justified.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool gets the current mouse cursor position and returns coordinates. The verb 'Get' and resource 'mouse cursor position' are specific, and it distinguishes from siblings like mouse_move (which sets position).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives (e.g., before clicking, after moving). No exclusions or context provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
double_clickA
Grounded double click in the most recent capture's pixel space. Pass source=ocr_center for an OCR-returned center; otherwise the reported route defaults to visual_coordinate.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| source | No | Grounding source for route reporting. Use `ocr_center` only for a center returned by the immediately preceding OCR result. Omit/default `visual_coordinate` for a focused screenshot/zoom coordinate. | |
| background | No | Deliver in the background to the captured window's process (native apps only; browsers/Electron ignore it). Default false = foreground. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It usefully discloses the coordinate space ('most recent capture's pixel space') that grounds the click, but says nothing about side effects, required permissions, failure when no capture exists, or whether the target window is brought to the foreground by default.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with the action and coordinate frame front-loaded, and the parameter routing appended second. No filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutating input tool with no annotations and no output schema, the description covers the coordinate-grounding model but omits prerequisites (an existing capture), side effects, and failure modes. Adequate but with clear gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: source and background are documented in the schema, while x and y are bare numbers. The description partially compensates by framing x/y as living in 'the most recent capture's pixel space' and clarifying the default route, but adds no format or range detail for the coordinates themselves.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource: a double click 'grounded in the most recent capture's pixel space', which is more informative than the bare name. It implicitly distinguishes itself from left_click/right_click by naming the double-click action, but never explicitly contrasts with those siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives conditional guidance for the source parameter ('pass source=ocr_center for an OCR-returned center; otherwise defaults to visual_coordinate'), which is closest to routing guidance. It does not say when to choose this tool over left_click/right_click or what preconditions (a prior capture) must hold.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
dump_axA
DEBUG: dump the target app's Accessibility tree (roles, subroles, labels, actions, frames) as indented text — to diagnose why some elements are not exposed. Targets the current AX/UIA app (or frontmost).
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It usefully discloses the output shape (indented text) and target resolution ('current AX/UIA app, or frontmost'), but never states that it is a non-mutating/read-only operation or any performance/cost characteristics, leaving the safety profile to inference.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, front-loaded sentence prefixed with DEBUG, using the em dash to separate the action from the diagnostic rationale. Every clause earns its place with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter debug tool with no output schema, the description supplies everything an agent needs: what is dumped, what fields it contains, the output format, and which app it resolves to.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so per the baseline there is nothing for the description to disambiguate. The mention that it targets the current AX/UIA app or frontmost is a small bonus, but the baseline of 4 applies for a 0-param schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('dump') and resource ('target app's Accessibility tree') and enumerates what the dump contains (roles, subroles, labels, actions, frames). The DEBUG framing and diagnosis intent clearly separate it from siblings like ax_read, read_ui, and inspect_app.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear context for use: diagnose why some elements are not exposed, distinguishing it from normal read/inspect tools. It does not state when NOT to use it (e.g., prefer ax_read for straightforward reading), so no explicit exclusions or named alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
inspect_appA
Inspect a running macOS application's interaction options, or discover running Chromium applications when app is omitted. Returns a concise identity, status, native route and next step; details=true adds diagnostics. Read-only: no app launch, focus, permission request, page content, or browser attachment. Use optionally for setup/discovery; ordinary native interaction still starts with ax_read.
| Name | Required | Description | Default |
|---|---|---|---|
| app | No | Running application name or bundle identifier. Omit to discover Chromium apps. | |
| details | No | Include bounded technical diagnostics. Default output is a concise summary. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so explicitly: it is read-only and enumerates what it will NOT do (no app launch, focus, permission request, page content, browser attachment). It also discloses the return shape, compensating for the absent output schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose in the first clause, then returns, then constraints, then usage in a single tight block. Dense semicolon-packed clauses are efficient, though slightly heavy for the amount of information conveyed.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Two optional parameters, no annotations, and no output schema, yet the description supplies the missing pieces: what it returns, what it abstains from, and how it relates to ax_read. An agent has everything needed to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both parameters are already documented; the description largely restates the omission semantics and the details=true diagnostics rather than adding syntax or format detail. Baseline 3 applies when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb ('Inspect') plus resource ('a running macOS application's interaction options') and an explicit second mode (discover Chromium apps when app is omitted). It names ax_read as the tool for ordinary native interaction, so an agent can separate it from the ax_* and chrome_* siblings without reading any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
States when to use it ('optionally for setup/discovery') and where the default path lies ('ordinary native interaction still starts with ax_read'), which routes the agent to the correct alternative. Nothing is left to inference about positioning.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
key_comboC
Simulate a key combination (e.g., "cmd+c", "shift+tab").
| Name | Required | Description | Default |
|---|---|---|---|
| key | Yes | ||
| background | No | Deliver in the background to the captured window's process (native apps only). Default false = foreground. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden, yet it never says where the keystrokes are delivered (focused window, active app, or captured process), whether the call blocks until delivery, or whether it fails silently. The examples convey accepted syntax but no runtime behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with an inline example pair; nothing is padded. It is efficient, though the brevity leaves the gaps noted in other dimensions.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an OS-level input tool with no annotations and no output schema, the description should explain delivery target and focus requirements. As written, an agent cannot tell what state the machine must be in for the combination to land correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: the 'background' parameter is self-documented, but 'key' is a bare string in the schema. The description's examples ("cmd+c", "shift+tab") supply the chord syntax that the schema omits, adding real value, but it doesn't cover key names or multi-modifier ordering.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb (simulate) and resource (key combination) and gives two concrete examples, which lets an agent distinguish it from type_text. It doesn't explicitly name the sibling it differs from, so it falls short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus type_text or batch_actions, and no mention of prerequisites such as the target window needing focus. Usage must be inferred entirely from the name and examples.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
left_clickA
Last-resort grounded left click in the most recent capture's pixel space. Pass source=ocr_center when x/y came from focused OCR; otherwise source=visual_coordinate (the default) means a focused screenshot/zoom. The result reports that route.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| source | No | Grounding source for route reporting. Use `ocr_center` only for a center returned by the immediately preceding OCR result. Omit/default `visual_coordinate` for a focused screenshot/zoom coordinate. | |
| background | No | Deliver in the background to the captured window's process (native apps only; browsers/Electron ignore it). Default false = foreground. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It usefully discloses the operating space ('most recent capture's pixel space') and that the result reports the grounding route, which is real behavioral context. But it says nothing about side effects of a click (focus changes, foreground vs background behavior), failure modes when no capture exists, or permissions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the core action and operating space before the parameter-routing guidance. Efficient, though the source explanation is dense and partly duplicates the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-parameter, no-annotation, no-output-schema action tool, the definition covers purpose, grounding source, and route reporting, but omits background behavior, click side effects, and what happens when no capture is available. Adequate but with clear gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%. The description explains source=ocr_center vs source=visual_coordinate, but that largely mirrors the schema's own source description, so it adds little beyond structured data. It adds no meaning for x/y (undocumented in both) and does not touch the background parameter, so it only partially compensates for the coverage gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: a left click executed in the most recent capture's pixel space, with the 'last-resort/grounded' framing signaling it is a fallback rather than a primary interaction. It implicitly separates itself from sibling ax_click/click_mark/chrome_* tools, but never names them, so the differentiation is inferred rather than explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Last-resort' gives clear ordering context (use only when other interaction paths fail), and the source guidance tells the agent how to pick the correct grounding route. However, no alternatives are named (e.g. ax_click, click_mark, chrome_click) and there is no explicit when-not condition.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_applicationsA
List all installed applications on the system.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It only states the tool lists installed applications, revealing no side effects, security implications, or performance characteristics. For a read-only listing, this is minimally adequate but lacks detail.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that efficiently communicates the tool's purpose without unnecessary words or structure issues.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no parameters, no output schema, and low complexity, the description is mostly complete. However, it omits what format the list returns (e.g., names only, objects), which could be clarified. Still sufficient for a simple list command.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has no parameters, and the schema coverage is 100% (empty schema). The description adds no parameter information because none is needed. Baseline 4 is appropriate as the description does not need to compensate for missing schema details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action 'list' and the resource 'all installed applications on the system', making the purpose immediately obvious. It is distinct from sibling tools like 'open_application' or 'list_windows'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives. The description implies usage for retrieving installed apps, but does not mention when not to use it or suggest alternatives for related tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_windowsA
List all visible windows across all applications.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose behavioral traits like read-only nature, speed, or permissions. It only states the basic function.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no wasteful content, efficiently conveying the tool's purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a parameterless tool with no output schema, the description adequately covers the core function. However, it could mention that the tool returns a list of window identifiers or details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist, and schema coverage is 100%. The description does not add extra semantics but is sufficient given the absence of parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists all visible windows across all applications, with a specific verb and resource, distinguishing it from sibling tools like list_applications.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as list_applications or click tools, nor any context on prerequisites or limitations.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
mouse_moveB
Move the mouse cursor to the given (x, y) coordinates (in screenshot space).
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden, and it discloses only that coordinates are in screenshot space. It says nothing about side effects (hover triggers, focus changes), whether the move is instantaneous or animated, permission requirements, or return behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single compact sentence with the key scoping detail ('in screenshot space') placed where it is read, and no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter, no-output-schema tool this is close to sufficient, but the coordinate frame is only loosely pinned down (no origin, no relation to screenshot resolution). An agent could call it but might mis-target if the screenshot was scaled.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate; it does define both x and y as coordinates in screenshot space, which is real added meaning. However it omits origin corner, units, and whether values relate to the screenshot's pixel dimensions or the live screen, leaving the semantics partial.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Move the mouse cursor') plus the coordinate target, and adds the important qualifier 'in screenshot space'. It does not differentiate itself from siblings like click_mark or left_click that also position the cursor, so it falls short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this versus alternatives such as click_mark, left_click, or ax_activate, and no stated prerequisites. The reader must infer usage entirely from the verb.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ocrA
Read on-screen TEXT via the platform OCR engine (Apple Vision on macOS, Windows.Media.Ocr on Windows). Captures the display (or window="") and returns the recognized text lines, each with a clickable center in the same pixel space as a screenshot — so you can both READ the text and click a line with left_click(x, y). Returns text only (no image), so it is a cheap, fast way to pull text off the screen. Best when use it after ax_read reports absent/partial coverage and you need rendered TEXT from a surface where semantic nodes are empty or sparse — canvas, games, image-rendered or custom-drawn views — or to grab a lot of text at once without parsing a screenshot. For native/web UI with an Accessibility/UIA tree, ax_read + ax_activate is still more precise. mode=auto (default) tries fast recognition first and only falls back to accurate for empty/low-confidence output; mode=fast minimizes latency and mode=accurate forces maximum quality. roi={x,y,width,height} scopes OCR in the CURRENT image's pixels: the rectangle is strictly validated, mapped to the desktop, and re-captured through the native region path (never cropped from a downscaled JPEG). roi and window are mutually exclusive. Languages default to Simplified Chinese + English; pass languages=[...] (BCP-47) to override.
| Name | Required | Description | Default |
|---|---|---|---|
| roi | No | Optional strict region of interest in the CURRENT image's pixel space. Nova maps it back to global logical coordinates and re-captures through the platform's native region path before JPEG encoding/OCR. It is never cropped from an already-downscaled JPEG. Every value must be finite, the size positive, and the entire rectangle in bounds. Mutually exclusive with `window`. | |
| mode | No | Recognition latency/quality policy. `auto` (default) starts with the fast Apple Vision recognizer and falls back to accurate only for empty or low-confidence output. On engines without a native mode knob the setting preserves that engine's normal behavior. | |
| window | No | Capture only a single on-screen window (case-insensitive substring of its title or app name) instead of the whole display. Smaller, sharper image → better recognition of small text. | |
| languages | No | BCP-47 language hints in priority order (e.g. ["zh-Hans", "en-US"]). Omitted, defaults to Simplified Chinese + English. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well: it discloses the return shape (text lines with clickable centers in screenshot pixel space), that no image is returned, the cheap/fast tradeoff, and the ROI re-capture path. It omits operational caveats like required screen-capture permissions and failure/error behavior, so it stops short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose and the cost/return profile are front-loaded, and the sentences are dense with operational detail. It runs long as a single block and repeats some schema content, but nearly every clause adds actionable information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no output schema, the description explains exactly what comes back (recognized lines with clickable centers usable via left_click) plus engine selection, modality, and param interactions. An agent has everything needed to call it and use the result correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds real meaning beyond it: the roi/window mutual exclusion, the strict validation and native re-capture of roi (never cropped from a downscaled JPEG), the auto/fast/accurate fallback policy, and the default language pair.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Read on-screen TEXT via the platform OCR engine') and immediately names the concrete engines. It also distinguishes itself from siblings by declaring it returns text only and routes UI-tree cases to ax_read + ax_activate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use ('after ax_read reports absent/partial coverage', canvas/games/image-rendered views, bulk text grabbing) and explicit when-not-to ('For native/web UI with an Accessibility/UIA tree, ax_read + ax_activate is still more precise'). Alternatives and selection conditions are both present.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
open_applicationA
Launch or focus an application by name (e.g., "Safari", "Slack").
| Name | Required | Description | Default |
|---|---|---|---|
| app | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description bears the full burden; it does disclose the meaningful launch-or-focus semantics (idempotent behavior). But it says nothing about behavior on an unknown app name, whether it blocks until the app is ready, permission requirements (e.g., accessibility/automation), or failure modes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single tight sentence with the action front-loaded and examples appended; every element earns its place with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter, no-output-schema tool this is close to minimally sufficient to invoke correctly. It still leaves the agent guessing about error behavior and name-matching rules, which matters for a launch action whose failure mode is user-visible.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single param has 0% schema description coverage, so the description must compensate; the inline examples ('Safari', 'Slack') usefully clarify that the parameter is a human-readable app name. It does not address case sensitivity, exact-name matching, or what identifier forms are accepted.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb pair (launch/focus) plus the resource (application) and the argument format ('by name'). It is clearly distinguishable from siblings like list_applications and inspect_app by virtue of being an action, though it never explicitly contrasts itself with app-specific activators such as chrome_activate or ax_activate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'launch or focus' implicitly signals this is safe to call whether or not the app is already running, which is useful usage context. However, it names no alternatives and gives no when-not guidance, so an agent facing chrome_activate / chrome_focus / ax_focus siblings gets no routing help.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
read_clipboardA
Read the current system clipboard contents as text.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden. 'Read' implies non-destructive, and the simplicity of the tool (no parameters) makes additional disclosure unnecessary. The description is adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with no wasted words. It is appropriately sized and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple, parameterless tool with no output schema, the description fully covers the intended behavior. No additional context is required.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, so schema coverage is 100%. The description adds no parameter info, but none is needed. Baseline score of 4 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Read') and resource ('system clipboard contents'), and specifies the format ('as text'). It clearly distinguishes from the sibling 'write_clipboard'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
While the purpose is clear, the description does not provide explicit guidance on when to use this tool versus alternatives, such as noting it is safe for reading without side effects.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
read_uiA
Compatibility alias for ax_read, backed by the exact same semantic traversal, renderer, snapshot generation, and action cache. Prefer the canonical ax_read name in new prompts and integrations.
| Name | Required | Description | Default |
|---|---|---|---|
| max | No | Cap on the number of elements returned (default 200, max 400). | |
| mode | No | `interactive` returns actionable controls, `content` returns readable labels/text, and `all` combines both in deterministic tree order. Defaults to `all`. | |
| filter | No | Show only elements whose role, label, or value contains this case-insensitive substring (e.g. "button", "search", "submit"). Omit to list everything. Filtering only narrows the DISPLAY — the hidden elements keep their numbers and stay clickable by click_mark(N). | |
| target | No | `native` (default) reads AX/UIA. `paired_page` reads only the already consented Chrome document; mutually exclusive with `window`. | |
| window | No | Target a single on-screen window: a case-insensitive substring of its title or app name (e.g. "Safari", "Settings"). Omit to read the current target app — the last window-captured or read_ui'd app, otherwise the frontmost. Web-page content is included either way (as long as the app has a window); a window= just narrows and sharpens the listing to that window. | |
| max_chars | No | Unicode-character output budget (default 30000, max 100000). If hit, `truncated=true` and `partial_reason=character_limit` are returned. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full disclosure burden, and it does add real value by guaranteeing behavioral equivalence to ax_read including the shared action cache. It does not mention safety profile (read-only vs mutating), side effects such as resetting the 'current target app' state implied by the window/target params, or output shape, leaving meaningful behavioral gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, zero padding, and the most important information (it is an alias for ax_read) is front-loaded before the migration advice.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a pure alias whose behavior is fully delegated to a named sibling and whose parameters are 100% documented in-schema, the description covers what an agent needs to select and call it. The absence of any return-value or side-effect note is a minor gap with no output schema present to compensate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% across all six parameters, with detailed inline docs for max, mode, filter, target, window, and max_chars. The description adds no parameter meaning beyond that, so the baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description identifies the tool as a compatibility alias of ax_read and lists what it shares (semantic traversal, renderer, snapshot generation, action cache), which lets an agent equate it with a sibling it likely already knows. However, it never plainly states the verb+resource itself (reading the accessibility/UI tree), so an agent unfamiliar with ax_read gets no direct statement of the action.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly directs the agent to prefer the canonical ax_read name in new prompts and integrations, which functions as a clear when-to-use-this-vs-alternative rule. It stops short of stating the inverse condition (when this alias is actually required, e.g. legacy integrations), but the routing decision is unmistakable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
right_clickB
Grounded right click in the most recent capture's pixel space. Pass source=ocr_center for an OCR-returned center; otherwise the reported route defaults to visual_coordinate.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| source | No | Grounding source for route reporting. Use `ocr_center` only for a center returned by the immediately preceding OCR result. Omit/default `visual_coordinate` for a focused screenshot/zoom coordinate. | |
| background | No | Deliver in the background to the captured window's process (native apps only; browsers/Electron ignore it). Default false = foreground. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It usefully discloses that the click is 'grounded' in the most recent capture's pixel space, a non-obvious behavioral constraint, but it says nothing about side effects (context menu), whether it requires a foreground window, or how it interacts with existing state.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with the core action front-loaded and the source-routing caveat second. No filler, though the second clause is dense and could be marginally easier to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-param mutation tool with no annotations and no output schema, the definition covers grounding and source routing but omits return behavior, the background-delivery implications, and alternative-selection context. Adequate but with clear gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: source and background are well documented in-schema, but x/y carry no schema description. The description partially compensates by establishing that coordinates live in 'the most recent capture's pixel space,' which is essential context for x/y, but gives no format, range, or origin details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('right click') and scopes it to 'the most recent capture's pixel space,' which clearly separates it from left_click, double_click, and ax_click. It does not explicitly name those siblings, but the action and grounding are unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The routing rule for source ('pass source=ocr_center for an OCR-returned center; otherwise defaults to visual_coordinate') is clear, but there is no guidance on when to choose right_click over ax_click, click_mark, or left_click, nor any prerequisite/exclusion context. Usage is implied rather than spelled out.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
screenshotA
Visual fallback after ax_read and, for missing rendered text, OCR. Capture the whole main display or a single window with window="" — and return a base64 JPEG plus a text note with its pixel dimensions. ALL coordinate-taking tools (mouse_move, *_click, scroll) expect coordinates in THIS image's pixel space — origin (0,0) top-left, x right, y down — so read target positions directly off the returned image; subsequent clicks are mapped through it automatically. PREFER window="" (substring of its title or app name) over the whole display whenever you are working inside one app: a full-display shot is downscaled to ~1280px wide, so small UI (list rows, sidebar items, buttons) becomes only a few pixels — too small to read or click accurately. A window capture is larger and sharper, returns a smaller image (fewer pixels → less context, faster turn), and clicks map into the window automatically. marks is ON by default: it boxes+numbers actionable elements (needs Accessibility) and lists each as [N] for compatibility with click_mark. Prefer a fresh ax_read + ax_activate for semantic controls. A magenta coordinate grid (for reading x/y) is shown automatically when marks is off and hidden when it is on; pass grid=true to force both, or marks=false for pure coordinate mode. If a target is still too small to click, use zoom_region to magnify part of this image.
| Name | Required | Description | Default |
|---|---|---|---|
| grid | No | Overlay a labeled coordinate grid (magenta rules + pixel labels) for reading off click coordinates. Omitted by default it follows the AX-first rule: OFF when `marks` is on (you click by number, not coordinates), ON when `marks` is off (coordinate mode). Pass grid=true to force it on alongside marks, or grid=false to suppress it. | |
| marks | No | Set-of-Mark: number every actionable UI element (buttons, links, fields) and list each as `[N] role "label"`, so you can activate it with click_mark(number=N) — the most reliable targeting, no coordinate guessing. Defaults ON (AX-first). Needs Accessibility permission. Covers native controls AND web content — real links/buttons on semantic pages, and on div-rendered pages (e.g. webmail) the list rows are numbered too (their click lands via a coordinate at the row center). Only canvas/game- style surfaces with no AX come back empty. Pass marks=false for pure coordinate mode. | |
| window | No | Capture only a single on-screen window instead of the whole display — a case-insensitive substring of the window title or app name (e.g. "Safari", "Settings"). Smaller, sharper image = less context and better click precision. Subsequent clicks map to this window automatically. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and does so: it discloses the return format (base64 JPEG plus a text note with pixel dimensions), the ~1280px downscale of full-display shots, that marks requires Accessibility permission, that clicks are mapped automatically through the captured image, and that canvas/game surfaces come back empty. This is exactly the kind of context annotations would otherwise supply.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the tool's role and the preferred argument, and every sentence carries actionable information (coordinate space, downscale, marks default, grid interplay, zoom fallback). It is a single dense block rather than a structured list, which slightly hurts scanability, but there is little waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having no output schema and no annotations, the description explains what is returned (image + dimension note), the coordinate space the image defines, the defaults for marks/grid, and the correct sibling tools for adjacent needs. An agent has everything required to call this correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents grid, marks, and window in depth; the baseline is 3. The description adds the practical rationale behind the defaults (the 'AX-first rule' toggling grid/marks, and why window= beats full-display), which is value beyond the schema, though it largely restates the schema's own descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb+resource (capture the whole main display or a single window) and immediately positions the tool relative to siblings: 'Visual fallback after ax_read and, for missing rendered text, OCR.' An agent can distinguish it from ax_read, ocr, and click_mark without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit routing guidance: use this as a visual fallback after ax_read/OCR, PREFER window=... when working inside one app (with the reason: full-display is downscaled to ~1280px), fall back to zoom_region if a target is still too small, and use marks for semantic controls via click_mark. Both when-to-use and when-not-to-use are covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrollC
Scroll at the given (x, y) position. Positive lines = up, negative = down.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| lines | Yes | ||
| background | No | Deliver in the background to the captured window's process (native apps only). Default false = foreground. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It does disclose the useful sign convention for lines (positive = up, negative = down), but says nothing about what x/y refer to (screen vs. window coordinates), whether the target must be focused, or what the background flag's 'native apps only' limitation means in practice.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with the core action front-loaded and the sign convention appended; nothing is wasted. It is arguably too terse given the undocumented parameters, but as a conciseness measure it is efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no annotations, no output schema, and three of four parameters undocumented in the schema, the description should supply more context. It never clarifies the coordinate system, target window handling, or the return/effect of the call, leaving meaningful gaps for a mutation-style input tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 25% (background alone is documented), so the description must compensate and it partially does: it explains the sign semantics of lines and frames x/y as a position. It leaves the coordinate space and units for x/y, and their interaction with the target window, undefined.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (scroll) and the position at which it occurs, so the action is unambiguous. It does not, however, differentiate itself from the near-identical sibling chrome_scroll, which an agent must distinguish on its own.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus chrome_scroll, chrome_activate, or other input tools, nor any mention of prerequisites or context such as focusing/targeting a window first. Usage is only weakly implied by the description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
type_textC
Type a string of text into the currently focused element.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| background | No | Deliver in the background to the captured window's process (native apps only). Default false = foreground. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It hints at a precondition ("currently focused element") but never says what happens if no element is focused, whether existing content is replaced or appended, or that accessibility/input permissions may be required. For an input-injection tool this is a substantial gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence, front-loaded with the verb and target, with no wasted phrasing. It is efficient, though its brevity is achieved by omission rather than precision.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no annotations, no output schema, and a focus-dependent input action, the description should cover failure modes and preconditions. It covers only the happy path, so an agent cannot predict behavior when focus is absent or when `background` is used with a non-native window.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: the optional `background` parameter is documented in the schema, but `text` has no schema description and the description only vaguely says "a string of text." It adds essentially nothing beyond the field names, leaving the foreground/background tradeoff (native apps only) unexplained in the description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
It states a specific verb ("Type") and resource ("a string of text") plus the target ("currently focused element"), which is enough to understand the operation. It does not distinguish itself from near-siblings such as ax_set_value, write_clipboard, or key_combo, which an agent must disambiguate on its own.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no when-to-use guidance and never mentions alternatives. It does not explain when to prefer typing over ax_set_value or clipboard-based insertion, nor what to do if nothing is focused. With ~30 sibling tools, this omission is significant.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
waitB
Wait for a specified number of seconds before returning.
| Name | Required | Description | Default |
|---|---|---|---|
| duration | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full burden. It hints at blocking behavior via 'before returning,' which is useful, but does not state whether the call is synchronous, any maximum duration, or consequences of long waits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no wasted words. It delivers the essential behavior immediately.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter, no-output tool this is nearly adequate, but it omits the default value, bounds, and return behavior that an agent would want before invoking a blocking wait.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema coverage, the description is the only source of meaning for the single 'duration' parameter, and it does supply the unit ('seconds') that the schema omits. However, it does not clarify the default of 1, allowed ranges, or whether fractional values are accepted.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (wait) and the resource/unit (seconds), so an agent immediately knows this is a blocking sleep. No sibling does anything similar, so sibling differentiation is unnecessary rather than missing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this versus alternatives (e.g., batch_actions for sequencing, or polling loops). The agent is left to infer that this is a delay primitive with no context about typical use cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
write_clipboardC
Write text to the system clipboard.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| background | No | Deliver in the background to the captured window's process (native apps only). Default false = foreground. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and falls short. It does not say whether the write overwrites or appends to existing clipboard content, what happens on failure or when the clipboard is unavailable, or whether it works in headless/non-native contexts.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single short, front-loaded sentence with zero waste. It is efficient, though the brevity edges toward under-specification for a tool with a non-obvious background flag.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter, no-output-schema tool, the description is minimal but workable. It leaves key behaviors (clipboard replacement semantics, background flag impact from the caller's perspective, failure modes) to inference, which is a modest gap for an operation that mutates shared system state.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: the background parameter is documented in the schema (background flow / native apps only), but the required text parameter has no description. The description's 'write text' implicitly maps to it but adds no format, length, or encoding detail beyond what the schema already implies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a concrete verb+resource pair: write to the system clipboard. It is unambiguous about what the tool does. However, it offers no differentiation from the sibling read_clipboard, so an agent gets no explicit routing signal beyond the obvious naming.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no when-to-use guidance and no mention of alternatives or the sibling read_clipboard. Usage can be inferred from the name, but the description provides nothing explicit about context, prerequisites, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
zoom_regionA
Zoom into a rectangle of the CURRENT image (the last screenshot's pixel space) and re-capture it at native resolution — a sharp, legible magnified view. Only that rectangle is captured (not the whole display), so it is also smaller and quicker to take than a full-display shot. Use it to read exact positions on surfaces that expose no marks (canvas/games, custom-rendered views) before clicking, or to stay scoped while working inside one area. Pass x, y, width, height in the current image's pixels (width,height > 0). The returned image becomes the new coordinate space, and clicks afterward map into the zoomed region automatically. marks defaults OFF (read coordinates off the overlaid grid); grid defaults ON. Take a screenshot first so there is an image to zoom into.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | Left edge of the rectangle, in the CURRENT image's pixel space (the last screenshot's coordinates). | |
| y | Yes | Top edge of the rectangle, in the current image's pixel space. | |
| grid | No | Overlay a labeled coordinate grid. Defaults ON for a zoom (coordinate mode); pass grid=false to suppress it. | |
| marks | No | Set-of-Mark numbering. Defaults OFF for a zoom — the zoom is the tool for surfaces that expose no marks, so you read coordinates off the grid. Pass marks=true to also number any actionable elements inside the region. | |
| width | Yes | Width of the rectangle in current-image pixels. Must be > 0. | |
| height | Yes | Height of the rectangle in current-image pixels. Must be > 0. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full burden and does so richly: only the rectangle is captured (not the whole display), it is smaller/quicker, the returned image becomes the new coordinate space, and subsequent clicks automatically map into the zoomed region. It also discloses the non-obvious defaults (marks off, grid on) and the reason behind them.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The core action and scope constraint are front-loaded, and nearly every sentence earns its place (caveats, defaults, coordinate-space behavior, prerequisite). Minor padding such as 'a sharp, legible magnified view' keeps it from a 5.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and no annotations, the description still explains what the returned image means (new coordinate space, automatic click remapping) and the prerequisite screenshot. Nothing an agent needs to call this correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3; the description adds value by framing x/y/width/height as current-image pixel space with width,height > 0 and by explaining the intent behind the grid/marks defaults. It does not add new syntax or ranges beyond the schema, so it stays just above baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource+scope: zoom into a rectangle of the CURRENT image and re-capture at native resolution, distinguishing it from a full-display screenshot in the same sentence. An agent can tell this apart from sibling 'screenshot' without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit when-to-use: read exact positions on surfaces that expose no marks (canvas/games, custom-rendered views) before clicking, or to stay scoped within one area. It also names the prerequisite (take a screenshot first) and the trade-off versus a full-display capture.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
30 tool updates
v0.3.0- Added
ax_activate - Changed
ax_click1 field changed- removed
Input schema / titleRemoved value: -"AxQueryParams"
- Changed
ax_focus1 field changed- removed
Input schema / titleRemoved value: -"AxQueryParams"
- Added
ax_read - Changed
ax_set_value1 field changed- removed
Input schema / titleRemoved value: -"AxSetValueParams"
- Changed
batch_actions3 fields changed- changed
Input schema / properties / actions / descriptionPrevious value: -"Ordered list of input actions to execute in a single call."New value: +"Up to 64 ordered input actions. Larger batches are rejected before any\naction runs. Execution stops at the first failure." - added
Input schema / properties / actions / maxItemsAdded value: +64 - removed
Input schema / titleRemoved value: -"BatchParams"
- Added
chrome_activate - Added
chrome_focus - Added
chrome_pair - Added
chrome_read - Added
chrome_release - Added
chrome_scroll - Added
chrome_set_value - Added
chrome_status - Changed
click_mark2 fields changed- changed
Input schema / properties / number / descriptionPrevious value: -"The mark number [N] of the element to activate, as listed by the most\nrecent `screenshot(marks=true)`."New value: +"The mark number [N] of the element to activate, as listed by the most\nrecent `read_ui` or `screenshot(marks=true)`." - removed
Input schema / titleRemoved value: -"ClickMarkParams"
- Changed
double_click3 fields changed- added
Input schema / $defsAdded value: +{ + "CoordinateActionSource": { + "oneOf": [ + { + "const": "ocr_center", + "description": "Center returned by the immediately preceding focused OCR result.", + "type": "string" + }, + { + "const": "visual_coordinate", + "description": "Coordinate read from the immediately preceding focused screenshot/zoom.", + "type": "string" + } + ] + } +} - added
Input schema / properties / sourceAdded value: +{ + "anyOf": [ + { + "$ref": "#/$defs/CoordinateActionSource" + }, + { + "type": "null" + } + ], + "description": "Grounding source for route reporting. Use `ocr_center` only for a center\nreturned by the immediately preceding OCR result. Omit/default\n`visual_coordinate` for a focused screenshot/zoom coordinate." +} - removed
Input schema / titleRemoved value: -"ClickParams"
- Added
inspect_app - Changed
key_combo1 field changed- removed
Input schema / titleRemoved value: -"KeyParams"
- Changed
left_click3 fields changed- added
Input schema / $defsAdded value: +{ + "CoordinateActionSource": { + "oneOf": [ + { + "const": "ocr_center", + "description": "Center returned by the immediately preceding focused OCR result.", + "type": "string" + }, + { + "const": "visual_coordinate", + "description": "Coordinate read from the immediately preceding focused screenshot/zoom.", + "type": "string" + } + ] + } +} - added
Input schema / properties / sourceAdded value: +{ + "anyOf": [ + { + "$ref": "#/$defs/CoordinateActionSource" + }, + { + "type": "null" + } + ], + "description": "Grounding source for route reporting. Use `ocr_center` only for a center\nreturned by the immediately preceding OCR result. Omit/default\n`visual_coordinate` for a focused screenshot/zoom coordinate." +} - removed
Input schema / titleRemoved value: -"ClickParams"
- Changed
mouse_move1 field changed- removed
Input schema / titleRemoved value: -"MouseMoveParams"
- Changed
ocr4 fields changed- added
Input schema / $defsAdded value: +{ + "OcrModeParam": { + "oneOf": [ + { + "const": "auto", + "description": "Fast first; retry accurately only for empty/low-confidence Vision output.", + "type": "string" + }, + { + "const": "fast", + "description": "Lowest latency, potentially lower recognition quality.", + "type": "string" + }, + { + "const": "accurate", + "description": "Highest quality, potentially slower.", + "type": "string" + } + ] + }, + "OcrRoiParams": { + "properties": { + "height": { + "description": "ROI height in current-image pixels; must be finite and > 0.", + "format": "double", + "type": "number" + }, + "width": { + "description": "ROI width in current-image pixels; must be finite and > 0.", + "format": "double", + "type": "number" + }, + "x": { + "description": "Left edge in the CURRENT image's pixel space.", + "format": "double", + "type": "number" + }, + "y": { + "description": "Top edge in the CURRENT image's pixel space.", + "format": "double", + "type": "number" + } + }, + "required": [ + "x", + "y", + "width", + "height" + ], + "type": "object" + } +} - added
Input schema / properties / modeAdded value: +{ + "anyOf": [ + { + "$ref": "#/$defs/OcrModeParam" + }, + { + "type": "null" + } + ], + "description": "Recognition latency/quality policy. `auto` (default) starts with the\nfast Apple Vision recognizer and falls back to accurate only for empty\nor low-confidence output. On engines without a native mode knob the\nsetting preserves that engine's normal behavior." +} - added
Input schema / properties / roiAdded value: +{ + "anyOf": [ + { + "$ref": "#/$defs/OcrRoiParams" + }, + { + "type": "null" + } + ], + "description": "Optional strict region of interest in the CURRENT image's pixel space.\nNova maps it back to global logical coordinates and re-captures through\nthe platform's native region path before JPEG encoding/OCR. It is never\ncropped from an already-downscaled JPEG. Every value must be finite, the\nsize positive, and the entire rectangle in bounds. Mutually exclusive\nwith `window`." +} - removed
Input schema / titleRemoved value: -"OcrParams"
- Changed
open_application1 field changed- removed
Input schema / titleRemoved value: -"OpenAppParams"
- Added
read_ui - Changed
right_click3 fields changed- added
Input schema / $defsAdded value: +{ + "CoordinateActionSource": { + "oneOf": [ + { + "const": "ocr_center", + "description": "Center returned by the immediately preceding focused OCR result.", + "type": "string" + }, + { + "const": "visual_coordinate", + "description": "Coordinate read from the immediately preceding focused screenshot/zoom.", + "type": "string" + } + ] + } +} - added
Input schema / properties / sourceAdded value: +{ + "anyOf": [ + { + "$ref": "#/$defs/CoordinateActionSource" + }, + { + "type": "null" + } + ], + "description": "Grounding source for route reporting. Use `ocr_center` only for a center\nreturned by the immediately preceding OCR result. Omit/default\n`visual_coordinate` for a focused screenshot/zoom coordinate." +} - removed
Input schema / titleRemoved value: -"ClickParams"
- Changed
screenshot1 field changed- removed
Input schema / titleRemoved value: -"ScreenshotParams"
- Changed
scroll1 field changed- removed
Input schema / titleRemoved value: -"ScrollParams"
- Changed
type_text1 field changed- removed
Input schema / titleRemoved value: -"TypeParams"
- Changed
wait1 field changed- removed
Input schema / titleRemoved value: -"WaitParams"
- Changed
write_clipboard1 field changed- removed
Input schema / titleRemoved value: -"TypeParams"
- Changed
zoom_region1 field changed- removed
Input schema / titleRemoved value: -"ZoomRegionParams"
23 tool updates
v0.1.0- First observed
ax_click - First observed
ax_focus - First observed
ax_set_value - First observed
batch_actions - First observed
click_mark - First observed
cursor_position - First observed
double_click - First observed
dump_ax - First observed
key_combo - First observed
left_click - First observed
list_applications - First observed
list_windows - First observed
mouse_move - First observed
ocr - First observed
open_application - First observed
read_clipboard - First observed
right_click - First observed
screenshot - First observed
scroll - First observed
type_text - First observed
wait - First observed
write_clipboard - First observed
zoom_region
TDQS
Scored across 35 tools
Two parallel semantic pipelines cover the same Chrome automation ground: ax_read/ax_activate with target=paired_page versus chrome_read/chrome_activate, with no clear rule for when to pick which. Clicking a UI element can also be done through ax_activate, chrome_activate, click_mark, or left_click, and legacy ax_click/ax_focus/ax_set_value plus the read_ui alias add further alternatives. Descriptions label preferred vs legacy vs compatibility routes, but the boundaries remain blurry enough to invite misselection.
There is a common verb_noun pattern in tools like read_clipboard, write_clipboard, click_mark, open_application, type_text, zoom_region, and inspect_app, and the ax_ and chrome_ families are internally consistent. However, many tools break that pattern: action-only names (scroll, wait, screenshot, ocr), noun_noun names (cursor_position, batch_actions, key_combo), and adjective_noun names (left_click, right_click, double_click). The set is readable but lacks one predictable convention.
At 35 tools, the server exceeds the well-scoped 3-15 range and sits in the too-many category. Several tools are explicitly legacy, compatibility, or debug variants (ax_click, ax_focus, ax_set_value, read_ui, click_mark, dump_ax), and the ax_* versus chrome_* Chrome pipelines duplicate capability rather than each earning a distinct place. The broad computer-use domain justifies a rich surface, but this set could be consolidated.
Coverage is strong across the apparent domain: native UI read/action, Chrome semantic lifecycle (pair, status, read, activate, focus, set value, scroll, release), visual fallbacks (screenshot, OCR, zoom), low-level input (mouse, keyboard, clipboard, scroll), and application/window discovery. Gaps such as drag-and-drop, hover, window manipulation, and conditional waits are real but minor and usually workable through existing primitives.
Maintenance
Related MCP Connectors
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Use your Mac, Windows or Linux computer from ChatGPT, Claude or Codex: files, commands, documents.
Real Chrome for agents: start a browser, read pages as numbered markdown, click, type, hand off.
- fastCRWOAuthio.github.us
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Related MCP Servers
- AlicenseAqualityCmaintenanceGive AI agents eyes and hands on macOS.2226 npm25MIT
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to see and control your desktop with tools for screenshots, clicks, typing, and more, all locally on macOS and Windows.180 npmMIT
- AlicenseAqualityDmaintenanceEnables AI agents to control macOS desktop apps via screenshots, mouse clicks, keyboard input, accessibility queries, and AppleScript.1111 npmMIT
- AlicenseNot gradedqualityBmaintenanceProvides a JSON-RPC computer use runtime for macOS, exposing 7 MCP tools (observe/act/inspect/session/cancel/trace) as image content blocks so external agents like Claude Code, Pi, OpenCode, or Codex CLI can capture screenshots and drive the desktop with clicks, keys, and typing while enforcing session locking, stale-frame protection, and trace redaction server-side.10 npm1MIT