Touchpoint
Touchpoint's MCP server gives LLM agents full read and control access to desktop UI elements across Linux, macOS, and Windows via native accessibility APIs (AT-SPI2, UIA, AX) and Chrome DevTools Protocol for browser/Electron web content.
Discovery & Inspection
apps— List all applications visible in the accessibility treewindows— List all open windows with IDs, titles, sizes, and app nameselements— Browse the accessibility tree with filtering by role, state, depth, and sort orderfind— Search for UI elements by name using 4-stage matching (exact → contains → word → fuzzy)get_element— Fetch a detailed snapshot of a single element by ID, including supported actions
Screenshots
screenshot— Capture the full desktop, a specific app, window, element bounding box, or monitor
UI Actions
click— Click an element by ID or screen coordinates (left, right, or double-click)set_value— Set text content of an editable fieldset_numeric_value— Set the value of a slider or spinboxfocus— Move keyboard focus to an elementaction— Execute any raw accessibility action by name (e.g. "expand or collapse")activate_window— Bring a window to the foreground
Keyboard & Mouse Input
type_text— Type text into the focused element (supports\n,\t,\b)press_key— Press a single key or combination (e.g.["ctrl", "s"]), with repeat supportmouse_move— Move the cursor to an element or screen coordinatesscroll— Scroll up, down, left, or right at the cursor position
Waiting / Polling
wait_for— Wait for UI elements to appear or disappear, with multi-query andany/allmodeswait_for_app— Wait for an application to appear or disappearwait_for_window— Wait for a window (matched by title substring) to appear or disappear
Output is returned in flat, tree, or JSON formats with persistent element IDs for reliable cross-action references.
Enables interaction with Discord's desktop interface by reading its structured accessibility data and Electron-based web content.
Provides deep integration with Electron-based applications, allowing AI agents to read and interact with both native and web-based UI elements via CDP.
Enables discovery and interaction with Firefox UI elements, allowing agents to automate browser tasks like filling fields and clicking buttons.
Supports automated interaction with Google Chrome through native accessibility APIs and Chrome DevTools Protocol (CDP) for structured access to web content.
Allows programmatic control over the Slack desktop application by reading its accessibility tree to locate and interact with UI components like buttons and message fields.
Touchpoint is a cross-platform Python library for reading and interacting with desktop UI through native accessibility APIs. One import, one API — works on Linux, macOS, and Windows, with built-in support for Chromium and Electron apps via CDP (Chrome DevTools Protocol).
Instead of scraping pixels or running vision models, Touchpoint reads the real accessibility tree — structured names, roles, states, and positions for every element on screen. Fast and reliable, with no vision model required. Ships with an MCP server so LLM agents like Claude, Cursor, or any local model can control any desktop app out of the box.
import touchpoint as tp
elements = tp.find("Send", role=tp.Role.BUTTON, app="Slack")
tp.click(elements[0])Why Touchpoint?
Screenshot / vision | Browser automation | Touchpoint | |
Native desktop apps | ⚠️ inaccurate or slow | ❌ | ✅ |
Browsers | ⚠️ inaccurate or slow | ✅ | ✅ via CDP |
Electron apps (Slack, VS Code, ...) | ⚠️ inaccurate or slow | ⚠️ web content only | ✅ native + web |
Structured element data | ❌ needs OCR/vision model | ✅ web only | ✅ names, roles, states, positions |
Works with local / non-vision models | ❌ | ✅ web only | ✅ all apps |
Works across Linux, macOS, Windows | ✅ | ✅ | ✅ |
Table of Contents
Related MCP server: servo-mcp
Install
Requires Python 3.10+.
pip install touchpoint-pyEverything is included: your platform's native backend, CDP support for browsers and Electron apps, the MCP server, and screenshot capabilities. Platform-specific dependencies are installed automatically via pip environment markers.
Platform requirements
Platform | Backend | Requirement |
Linux | AT-SPI2 | Install |
Windows | UI Automation | None — uses built-in COM APIs |
macOS | Accessibility (AX) | Grant permission: System Settings → Privacy & Security → Accessibility |
Quick Start
import touchpoint as tp
# Discover
apps = tp.apps() # ["Firefox", "Slack", "Terminal", ...]
windows = tp.windows() # Window objects with title, position, size
all_els = tp.elements(app="Firefox", named_only=True) # only elements with text labels
# Find
results = tp.find("Search", role=tp.Role.TEXT_FIELD, app="Firefox")
# Act
tp.set_value(results[0], "touchpoint python", replace=True)
tp.press_key("enter")
tp.hotkey("ctrl", "s") # keyboard shortcuts
# Wait for UI changes
tp.wait_for("results", app="Firefox", timeout=10)
# Screenshot
img = tp.screenshot() # full desktop → PIL.Image
img = tp.screenshot(app="Firefox") # cropped to app windowElement IDs
Every element has a unique ID like atspi:1234:1:2.0 or cdp:9222:TID:4. Action functions accept either an Element object or a bare ID string — useful for storing references across steps:
results = tp.find("Send", max_results=1)
element_id = results[0].id # "atspi:1234:1:5.2"
# later...
tp.click(element_id) # works with just the stringOutput formats
Control how results are returned:
tp.elements(app="Slack", format="flat") # one compact line per element (best for LLMs)
tp.elements(app="Slack", format="tree") # indented parent/child hierarchy
tp.elements(app="Slack", format="json") # full JSON with all fieldsMCP Server
Touchpoint ships an MCP (Model Context Protocol) server ready for any MCP-compatible client. Use it to let LLM agents like Claude, Cursor, local models, or any tool that supports MCP control your desktop.
Two modes — vision and no-vision
Set TOUCHPOINT_MODE=no-vision (default: vision) to switch modes:
Vision mode — agents use
screenshot()to see the screen and interact by element ID or coordinates. Best for frontier models with strong vision capabilities.No-vision mode — agents use
snapshot()to get a compact structured text tree of the active window, then act on element IDs directly. Works with any model including local ones that have no vision capability. Most action tools append auto-verify flags ((new window: ...),(focus moved),(no change detected)) so the agent can detect state changes without taking a screenshot.
Tools
Category | Vision mode | No-vision mode |
Orient |
|
|
Find |
|
|
Read |
|
|
Actions |
|
|
Keyboard |
|
|
Mouse |
|
|
Window |
|
|
Waiting |
|
|
Health |
|
|
The MCP server includes built-in instructions that teach agents the correct workflow for each mode — including the orient → act → verify loop, when to use read_text vs find, and how to recover from errors.
┌──────────┐
┌───▶│ ORIENT │ screenshot · apps · windows
│ └────┬─────┘
│ ▼
│ ┌──────────┐
│ │ LOCATE │ find · snapshot · get_element
│ └────┬─────┘
│ ▼
│ ┌──────────┐
│ │ ACT │ click · set_value · type_text · press_key
│ └────┬─────┘
│ ▼
│ ┌──────────┐
│ │ VERIFY │───▶ Done ✅
│ └────┬─────┘
│ │ not yet
└─────────┘Client setup
Config file location:
macOS:
~/Library/Application Support/Claude/claude_desktop_config.jsonWindows:
%APPDATA%\Claude\claude_desktop_config.json
{
"mcpServers": {
"touchpoint": {
"command": "touchpoint-mcp"
}
}
}If using a virtualenv, use the full path: "/path/to/venv/bin/touchpoint-mcp"
Add to .vscode/mcp.json in your workspace:
{
"servers": {
"touchpoint": {
"command": "touchpoint-mcp"
}
}
}Create or edit ~/.cursor/mcp.json:
{
"mcpServers": {
"touchpoint": {
"command": "touchpoint-mcp"
}
}
}Edit ~/.codeium/windsurf/mcp_config.json:
{
"mcpServers": {
"touchpoint": {
"command": "touchpoint-mcp"
}
}
}claude mcp add touchpoint -- touchpoint-mcpAdd to mcpServers in ~/.openclaw/openclaw.json:
{
"mcpServers": {
"touchpoint": {
"command": "touchpoint-mcp"
}
}
}Environment variables
Variable | Example | Description |
|
| Auto-discover CDP ports from running processes |
|
| Explicit app-to-port mapping (JSON) |
|
| Single app name (pair with |
|
| Single port (pair with |
|
| Seconds between CDP port scans |
|
| Display scale override |
|
| Minimum match score for find() (0.0–1.0) |
|
| Use coordinate fallback when native actions fail |
|
| Maximum elements per query |
|
| Default tree depth limit |
|
| Max seconds to wait for a macOS AX app reply |
Browser & Electron Apps (CDP)
Native accessibility APIs return limited data for Electron and Chromium apps (Slack, Discord, VS Code, etc.). Touchpoint's CDP backend connects via Chrome DevTools Protocol to get the full web content.
Auto-discovery is enabled by default — Touchpoint automatically finds running browsers and Electron apps that were launched with a debug port. No manual configuration needed beyond launching the app with the flag.
Setup
Launch the app with a debug port:
# Linux
google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/tp-chrome
# macOS
open -na "Google Chrome" --args --remote-debugging-port=9222 --user-data-dir=/tmp/tp-chrome
# Windows
start chrome --remote-debugging-port=9222 --user-data-dir=%TEMP%\tp-chromeConfigure Touchpoint:
import touchpoint as tp
tp.configure(cdp_discover=True) # auto-discover from running processes
# or
tp.configure(cdp_ports={"Google Chrome": 9222}) # explicit mappingControl what you get with the
sourceparameter:
tp.elements(app="Google Chrome", source="full") # native chrome + web content (default)
tp.elements(app="Google Chrome", source="cdp_ax") # web content only (CDP accessibility tree)
tp.elements(app="Google Chrome", source="native") # native UI only (toolbar, tabs, menus)
tp.elements(app="Google Chrome", source="dom") # DOM walker (catches what AX misses)CDP results are merged with native backend results — you get the toolbar and window controls from AT-SPI2/UIA/AX, combined with the full web page content from CDP, in a single elements() call.
source="ax" remains accepted as a compatibility alias for
source="cdp_ax". Prefer cdp_ax in new code so it is not confused with
the native macOS AX backend.
API Reference
Discovery
Function | Description |
| List application names in the accessibility tree |
| All windows with id, title, app, position, size, active state |
| UI elements, with filtering, tree mode, and formatting |
| Deepest element at screen coordinates |
| Fresh snapshot of a single element by ID |
Search & Wait
Function | Description |
| Search by name — 4-stage matching: exact → contains → word → fuzzy |
| Poll until elements appear (or disappear with |
| Poll until an app appears or disappears |
| Poll until a window appears or disappears |
Actions
Function | Description |
| Click via accessibility action, with coordinate fallback |
| Double-click |
| Right-click / context menu |
| Set text content ( |
| Set slider or spinbox value |
| Select a substring within text content across Linux, Windows, macOS, and web/CDP |
| Select a character range when you already know the offsets |
| Move keyboard focus |
| Execute a raw accessibility action by name |
| Bring a window to the foreground (restores from minimized) |
| Minimize a window. Use |
| Enter or exit fullscreen for a window |
| Politely close a window |
| Move a window to a new screen position |
| Resize a window to width × height pixels |
Input
Function | Description |
| Type into the currently focused element |
| Press and release a key ( |
| Key combination ( |
| Click at screen coordinates |
| Double-click at coordinates |
| Right-click at coordinates |
| Move the cursor |
| Scroll at current cursor position |
Screenshot & Config
Function | Description |
| Full desktop or cropped to app/window/element/monitor |
| Number of connected monitors |
| Set runtime options (see Configuration) |
| Report backend, input, CDP, timeout, and dependency health |
All action functions accept an Element object or a string ID. elements(), find(), and get_element() support format="flat", format="json", or format="tree" (elements only) to return pre-formatted strings instead of objects. Window management is implemented across Linux AT-SPI2, Windows UIA, and macOS AX backends.
Architecture
┌───────────────────────────────────────────────────────┐
│ import touchpoint as tp │
│ tp.find() · tp.click() · tp.screenshot() · ... │
│ (Public API) │
├─────────────────────────┬─────────────────────────────┤
│ Backend (ABC) │ InputProvider (ABC) │
├─────────────────────────┼─────────────────────────────┤
│ AT-SPI2 (Linux) │ Xdotool (X11) │
│ UIA (Windows) │ SendInput (Win32) │
│ AX (macOS) │ CGEvent (macOS) │
│ CDP (browsers) │ │
├─────────────────────────┴─────────────────────────────┤
│ Utilities: formatter · matcher · screenshot · scale │
└───────────────────────────────────────────────────────┘Two-layer design:
Backend reads the accessibility tree and runs structured actions (click, set_value, focus). Element-aware and reliable.
InputProvider simulates raw keyboard and mouse input. Coordinate-based and element-blind. Used as an automatic fallback when a native accessibility action isn't available.
CDP runs alongside the platform backend. Their results are merged: native window chrome (toolbar, tabs, menus) from AT-SPI2/UIA/AX, plus full web content from CDP, unified under one API.
For detailed internals, see ARCHITECTURE.md.
Configuration
tp.configure(
fuzzy_threshold=0.6, # minimum match score for find() (0.0–1.0)
fallback_input=True, # use InputProvider when native actions fail
type_chunk_size=40, # split long text into chunks for typing (0 = disable)
max_elements=5000, # max elements per query
max_depth=20, # default tree depth limit
scale_factor=None, # display scale override (None = auto-detect)
cdp_ports={"Chrome": 9222}, # explicit CDP port mapping
cdp_discover=True, # auto-discover CDP ports from running processes
cdp_refresh_interval=5.0, # seconds between CDP target scans
ax_messaging_timeout=1.0, # max seconds to wait for a macOS AX app reply
)tp.diagnostics() returns a JSON-friendly health report. It includes the
active backend, input provider, CDP targets, optional platform tools, configured
timeouts, and macOS apps recently skipped after an AX messaging timeout.
Development
git clone https://github.com/Touchpoint-Labs/touchpoint.git
cd touchpoint
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytestStatus
Alpha — fully functional and tested on all three platforms. The API may change before 1.0 based on user feedback.
Platform | Backend | Input | CDP | Tests |
Linux (X11) | ✅ AT-SPI2 | ✅ xdotool | ✅ | ✅ |
Windows | ✅ UIA | ✅ SendInput | ✅ | ✅ |
macOS | ✅ AX | ✅ CGEvent | ✅ | ✅ |
Known limitations
Wayland input — The Linux InputProvider uses
xdotool, which requires X11. On pure Wayland (no XWayland), keyboard/mouse simulation is unavailable. The accessibility tree and native actions still work.Synchronous CDP — CDP calls block on WebSocket responses. JavaScript dialogs (alert, confirm, prompt) are auto-dismissed to prevent deadlocks. An async rewrite is planned.
No browser navigation API — Touchpoint doesn't have built-in URL navigation. Agents can navigate by interacting with UI elements directly: find the address bar, type a URL, press Enter.
CDP windows are page targets, not OS windows — but window management still works:
tp.activate_window()brings the target forward via CDP, andminimize/fullscreen/close/move/resizeon a surfacedcdp:window are routed to the underlying native OS window (resolved by owning PID) and handled by the platform backend. They raiseActionFailedErroronly if no native OS window for that target can be found (e.g. it has been closed).Backend role/state parity is still uneven — macOS AX and Windows UIA both improved significantly in
0.3.0, but Windows still relies on more heuristics and has more unmapped long-tail roles than the other backends.
Roadmap
High Priority
Async CDP architecture — non-blocking WebSocket, proper dialog queuing, concurrent multi-tab queries
Medium Priority
Backend role/state parity — close remaining role mapping gaps, especially UIA long-tail roles on Windows
Wayland input backend —
libei/xdg-desktop-portalRemoteDesktop when X11 isn't available
Lower Priority
Tooltip and notification visibility
Element caching
License
Available Tools
27 toolsactionA
Perform a raw accessibility action by exact name.
Only call this when the element shows actions=[...] in snapshot()
output AND click() either failed silently or is not the right verb.
Use the exact action name shown in the snapshot — do not guess.
Example: an element showing actions=[expand, collapse] can be acted
on with action(element_id, "expand").
Args:
element_id: The element ID.
action_name: Exact action name from the snapshot actions=[...] list.
Inputs: element_id (required), action_name (required). Returns: result
activate_windowA
Bring a window to the foreground.
Element-scoped actions (click, set_value, select_text, etc.) bring
their window to focus automatically — you rarely need this. Use it
when you need to type_text or press_key in a window without first
clicking an element in it (e.g. sending a keyboard shortcut to a
window you haven't interacted with yet in this task).
Args:
window_id: The window ID to activate (from windows()).
Inputs: window_id (required). Returns: result
appsA
List applications with accessible UI elements.
Returns application names visible in the accessibility tree. Use these names to scope other tools (find, elements, screenshot).
Inputs: no named parameters. Returns: result
clickA
Click an element by ID, or at screen coordinates.
Pass ``element_id`` to click via the element's native
accessibility action (most reliable). Pass ``x`` and ``y``
to click directly at screen coordinates instead — useful
when clicking by ID triggers an unintended action (e.g.
opens a dropdown instead of focusing a text entry).
Every element shows its position as @(x,y) in listings.
Coordinate clicks always report OK even if nothing was hit —
verify the result with a screenshot or find().
Args:
element_id: The element ID to click.
x: Screen X coordinate (use with y instead of element_id).
y: Screen Y coordinate (use with x instead of element_id).
button: "left" (default) or "right".
double_click: If true, perform a double-click instead.
Cannot be combined with button="right".
Inputs: element_id, x, y, button, +1 more. Returns: result
close_windowB
Close a window.
Use windows() to find the window ID first.
Args:
window_id: The window ID to close.
Inputs: window_id (required). Returns: result
diagnosticsA
Report backend, input, CDP, and optional dependency health.
Args:
probe: If true (default), initialise optional providers and perform
discovery. Pass false for a side-effect-free state snapshot.
Inputs: probe. Returns: result
findA
Search for UI elements by name — buttons, links, labels, fields.
Matches against element names (button labels, link text, field
labels), NOT body text or prose content inside articles or
documents.
Returns element IDs that you can use with click, set_value, etc.
Use the FULL visible text for best results (e.g. "Send Message"
not just "Send").
Args:
query: Text to search for (e.g. "Send Message", "Submit", "Search").
app: Scope to this application (e.g. "Firefox", "Slack").
window_id: Scope to this window.
role: Only match this role (e.g. "button", "text_field", "link").
states: Only match elements with ALL these states (e.g. ["enabled", "visible"]).
max_results: Maximum matches to return.
fields: Which fields to search -- ["name"], ["name", "value"], or ["name", "value", "description"].
source: "full" (default, merged native+web), "cdp_ax" (CDP accessibility tree only), "native" (platform only), or "dom" (live DOM). "ax" remains as a compatibility alias for "cdp_ax".
Inputs: query (required), app, window_id, role, +4 more. Returns: result
focusB
Move keyboard focus to an element.
Args:
element_id: The element ID to focus.
Inputs: element_id (required). Returns: result
fullscreen_windowA
Enter or exit fullscreen for a window.
Use windows() to find the window ID first.
Args:
window_id: The window ID.
fullscreen: True to enter fullscreen (default), False to exit.
Inputs: window_id (required), fullscreen. Returns: result
get_elementA
Get a single element by its ID with full detail.
Returns a fresh snapshot with current states, value, supported
actions, and description. Use this to inspect an element
before calling the ``action()`` tool — the actions list shows
exactly which raw action names are available.
Args:
element_id: The element ID (from find/elements results).
Inputs: element_id (required). Returns: result
minimize_windowA
Minimize a window to the dock/taskbar.
Use windows() to find the window ID first.
To restore a minimized window, use activate_window().
Args:
window_id: The window ID to minimize.
Inputs: window_id (required). Returns: result
mouse_moveA
Move the mouse cursor to an element or to screen coordinates.
Use this before scroll() to scroll within a specific area.
Args:
element_id: The element ID to move the cursor to.
x: Screen X coordinate (use with y instead of element_id).
y: Screen Y coordinate (use with x instead of element_id).
Inputs: element_id, x, y. Returns: result
move_windowB
Move a window to a new screen position.
Use windows() to find the window ID first.
Args:
window_id: The window ID to move.
x: New horizontal position of the top-left corner.
y: New vertical position of the top-left corner.
Inputs: window_id (required), x (required), y (required). Returns: result
press_keyA
Press a key or key combination.
Single key: "enter", "tab", "escape", "f5", "backspace".
Combination: ["ctrl", "s"], ["ctrl", "shift", "p"], ["alt", "f4"].
Args:
keys: A single key name, or a list of keys for a combination
(all held together, then released in reverse order).
repeat: Number of times to press (default 1).
Inputs: keys (required), repeat. Returns: result
read_textA
Return text content exposed by an element or container.
Some backends expose aggregate descendant text for containers.
Pass a container ID from snapshot() to read an entire section,
article, dialog, or document body when the backend supports it.
Workflow: snapshot() to find the right container, read_text(id)
to read its full contents.
Args:
element_id: The element or container ID to read text from.
Inputs: element_id (required). Returns: result
resize_windowA
Resize a window.
Use windows() to find the window ID first.
Args:
window_id: The window ID to resize.
width: New width in pixels.
height: New height in pixels.
Inputs: window_id (required), width (required), height (required). Returns: result
screenshotA
Capture the screen and return an image.
With no arguments, captures the full desktop. Specify one
parameter to crop to a specific target.
Args:
app: Crop to this application's window.
window_id: Crop to this specific window.
element_id: Crop to this element's bounding box.
padding: Extra pixels around the crop region.
monitor: Capture only this monitor (0-indexed).
Inputs: app, window_id, element_id, padding, +1 more.
scrollA
Scroll by mouse wheel ticks.
With ``element_id``, moves the cursor to that element's position
first, then scrolls — useful for scrolling inside a specific list,
panel, or container. Without it, scrolls at the current cursor
position.
Args:
direction: One of "up", "down", "left", "right".
amount: Number of scroll ticks (default 3).
element_id: Optional element to scroll at. When given, cursor
is moved to the element's center first.
Inputs: direction (required), amount, element_id. Returns: result
select_textA
Select a substring within an element's text content.
Finds the text within the element and applies a native text
selection over that range. Works on editable fields and document
bodies on all backends. On web content (CDP) and Windows (UIA)
also works on read-only containers such as articles and sections.
Useful for formatting, copying, or replacing specific text.
Args:
element_id: The element ID containing the text.
text: The exact substring to select.
occurrence: Which occurrence to select (1 = first, 2 = second, etc.).
Inputs: element_id (required), text (required), occurrence. Returns: result
set_numeric_valueA
Set the numeric value of a range element (slider, spinbox).
Args:
element_id: The element ID (a slider, spin button, etc.).
value: The numeric value to set.
Inputs: element_id (required), value (required). Returns: result
set_valueA
Set the value of an editable element.
Calls the platform's native value-setting API. Behaviour depends
on the target widget:
* Text fields / text areas: writes the text.
* Combo boxes / dropdowns / <select>: picks the option whose
label or value matches *value*. Prefer this over clicking
the dropdown and then clicking a popup item — picking by name
is one atomic call and avoids popup-click failures on some
toolkits.
* Other editable widgets: whatever their value interface accepts.
Args:
element_id: The element ID (a text field, combo box, etc.).
value: The text or option label to set.
replace: If true, clear the field first and replace all content.
If false (default), insert at the current cursor position.
Inputs: element_id (required), value (required), replace. Returns: result
snapshotA
Return a tree-structured view of a window's UI.
The primary orient tool in no-vision mode. In vision mode, use it
as a structural complement to screenshot() — cheaper than a
screenshot and gives element IDs directly. Default behaviour: pick
the currently active window, walk its accessibility tree, prune
anonymous structural wrappers, preserve semantic containers (dialogs,
menus, lists, etc.), and emit an indented text view with one line per
interactive element.
Args:
app: Snapshot the given app's active window (or first window).
Case-insensitive.
window_id: Snapshot a specific window by ID.
element_id: Start the tree walk from a specific element instead
of the window root. Use to dig into a container whose
children were not visible in a previous (truncated) snapshot.
To read the text content of a container, use read_text()
rather than snapshot().
all_elements: If true, include every named element — not just
interactive + container roles. Use when the default
filter is hiding something.
max_depth: Maximum tree depth to walk. Defaults to the
configured value (typically 20). Decrease for a faster
overview of a large window.
Inputs: app, window_id, element_id, all_elements, +1 more. Returns: result
type_textA
Type text into the currently focused element.
Simulates keyboard input. Focus a text field first with
click() or focus(), then type into it.
Special characters:
\n = Enter (line break), \t = Tab (next field),
\b = Backspace (delete previous character).
Args:
text: The text to type.
raw: If true, type literal backslashes without converting
``\n``, ``\t``, or ``\b`` escape sequences.
Inputs: text (required), raw. Returns: result
wait_forA
Wait for elements to appear or disappear.
Polls until matching elements are found (or gone) or timeout.
Use after actions that trigger UI changes.
Args:
element: Text to search for. Pass a single string (e.g.
"Submit") or a list of strings (e.g. ["Success", "Error"])
for multi-query mode. With mode="any", returns as soon
as any query matches. With mode="all", waits until every
query has matched.
app: Scope to this application.
window_id: Scope to this window.
role: Only match this role.
states: Only match elements with ALL these states.
fields: Which fields to search (default: ["name"]).
mode: "any" (return when any query matches) or "all"
(wait for all queries to match). Only meaningful when
element is a list.
timeout: Maximum seconds to wait (default 10).
source: "full" (default), "cdp_ax", "native", or "dom".
"ax" remains as a compatibility alias for "cdp_ax".
max_results: Maximum elements to return (default 5).
wait_for_new: If true, ignore elements already present -- wait for NEW ones.
gone: If true, wait for matching elements to DISAPPEAR instead.
Inputs: element (required), app, window_id, role, +8 more. Returns: result
wait_for_appA
Wait for an application to appear or disappear.
Polls the application list until the app is found (or gone).
Use after launching or closing an application.
Args:
app: Application name to wait for (e.g. "Firefox", "Slack").
timeout: Maximum seconds to wait (default 10).
gone: If true, wait for the app to DISAPPEAR instead.
Inputs: app (required), timeout, gone. Returns: result
wait_for_windowA
Wait for a window to appear or disappear.
Polls the window list until a window with a matching title is
found (or gone). Use after actions that open or close windows.
When found, auto-activates the window so subsequent type_text
and press_key target it.
Args:
title: Window title to search for (substring match).
app: Only look for windows in this application.
timeout: Maximum seconds to wait (default 10).
gone: If true, wait for the window to DISAPPEAR instead.
Inputs: title (required), app, timeout, gone. Returns: result
windowsA
List all open windows.
Returns window IDs, titles, sizes, and app names.
Use window IDs to scope find/elements queries or activate_window.
Inputs: no named parameters. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 27 tools
Each tool targets a distinct resource and action: discovery (apps, windows, find, snapshot, screenshot), element inspection (get_element), input (click, set_value, set_numeric_value, focus, select_text, type_text, press_key), and window management (activate, minimize, close, move, resize, fullscreen). Overlaps like find vs wait_for are differentiated by polling behavior, and mouse_move/scroll have distinct roles.
All tools follow a consistent lower_snake_case verb_noun pattern (mouse_move, minimize_window, activate_window, set_value, wait_for_window, etc.). Exceptionally uniform, with no mixed conventions or vague verbs. Even atypical names like 'find', 'snapshot', and 'action' function as clear verbs.
At 27 tools, the count exceeds the typical 3-15 range, but the domain (cross-platform UI automation) justifies the breadth. Each tool covers a distinct operation—discovery, inspection, input, window manipulation, and waiting—with no redundancy. The count feels justified by the complexity required to handle various platforms and web content.
The surface is remarkably complete for UI automation: it covers discovery (apps/windows/elements), inspection (snapshot, get_element, read_text), interaction (click, type, set_value, scroll, drag via coordinate/action), and synchronization (wait_for, wait_for_app, wait_for_window). The generic `action` tool covers any raw accessibility action, providing a fallback for unforeseen operations. No critical dead-ends.
Maintenance
Related MCP Connectors
Cloud desktops for AI agents: create a Linux computer, see its screen, click, type and run tasks.
AI-powered browser automation — navigate, click, fill forms, and extract data from any website.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
A cloud Linux desktop for your AI: browse, click, type, read the screen and run commands.
Related MCP Servers
AlicenseBqualityBmaintenanceA Model Context Protocol server that enables LLMs to interact with web pages through structured accessibility snapshots without requiring vision models or screenshots.62510,815,609 npm39,637Apache 2.0- AlicenseNot gradedqualityDmaintenanceEnables AI agents to see and control your desktop with tools for screenshots, clicks, typing, and more, all locally on macOS and Windows.107 npmMIT
- AlicenseAqualityBmaintenanceDrive, inspect, and assert on real Electron desktop apps from an AI agent — agent-native, Playwright-style automation with accessibility refs, stable error codes, and retrying assertions5712MIT
- AlicenseNot gradedqualityBmaintenanceLets AI agents see and control desktop applications through the accessibility layer, enabling clicking, typing, scrolling, dragging, and window/app management across macOS, Windows, and Linux entirely on the local machine.25MIT