Skip to main content
Glama
Sign In
computer-use

computer-use

by Cassius-stack
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Local

window_action

Perform clicks, typing, scrolling, dragging, and element actions on macOS windows and apps, including background operations that do not move the user's cursor.

Instructions

Perform an action on a macOS window or app (click, hover, move, set_value, type_text, press_key, paste with automatic clipboard restore, scroll, drag, select_text, perform_secondary_action) with smooth 60fps virtual cursor feedback. Element-index actions operate in the background without moving the user's physical cursor.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "action": "<action>"
    },
    "name": "window_action"
  }
}

Input Schema

NameRequiredDescriptionDefault
xNoWindow-relative X coordinate (matches 1:1 window screenshot & AX @(x,y)).
yNoWindow-relative Y coordinate (matches 1:1 window screenshot & AX @(x,y)).
appNoTarget app name or bundle ID.
keyNoKey or chord (e.g. 'Return', 'cmd+s', 'ctrl+shift+p') for press_key or modified click.
textNoText for type_text, paste, or select_text.
to_xNoDrag end window-relative X.
to_yNoDrag end window-relative Y.
badgeNoOptional custom pill label next to the virtual cursor.
pagesNoScroll distance in pages (default 1).
valueNoValue string for set_value.
actionYes
formatNoPaste format (default 'text').
from_xNoDrag start window-relative X.
from_yNoDrag start window-relative Y.
prefixNoDisambiguating prefix for select_text.
rippleNoEmit click-style ripple ring on hover/move (default false).
suffixNoDisambiguating suffix for select_text.
directionNoScroll direction.
window_idNoTarget window ID.
click_countNo
duration_msNoHold duration in milliseconds.
show_cursorNoAnimate the per-window virtual cursor & highlight overlay for this action (default true).
mouse_buttonNo
return_stateNoAutomatically return updated AX state diff after action (default true).
window_titleNoOptional window title substring.
element_indexNoElement index [N] from get_window_state (preferred).
include_cursorNoInclude virtual cursor overlay in returned screenshot (default false).
selection_typeNo
activate_windowNoActivate target window before coordinate HID events (default false).
coordinate_onlyNoForce pure CGEvent coordinate dispatch on (x, y) click without Accessibility hit-test fallback (default false).
secondary_actionNoAX action name (e.g. 'ShowMenu', 'Increment', 'Decrement', 'Expand', 'Collapse').
to_element_indexNoOptional drag destination element index [N].
from_element_indexNoOptional drag start element index [N].
global_coordinatesNoInterpret (x, y) / drag / scroll coordinates as global screen points instead of window-relative points (default false).
include_screenshotNoInclude updated window screenshot (default false).
dispatch_move_eventNoFor hover/move, post a background mouseMoved event to the app PID (default true).

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv3.0.0

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full load and does disclose meaningful behavior: 60fps virtual cursor feedback, background element-index dispatch that leaves the user's physical cursor untouched, and paste with automatic clipboard restore. It still omits side-effect/risk disclosure an agent would want for a tool that types and clicks on the real machine, such as accessibility permission requirements and what happens if the target window is not frontmost.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the core verb and then the mode distinction. The parenthetical action list is long but replaces what would otherwise require reading an 11-value enum, so it earns its space; slightly dense but not padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 36-parameter mutation tool with no output schema, the description is decent but incomplete: it says nothing about return values despite return_state and include_screenshot defaulting to true, and does not mention that macOS accessibility/screen-recording permissions are required for cursor and HID actions. Those gaps matter for an agent deciding whether the call can succeed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 89%, so the schema already explains nearly every parameter (coordinates, element_index, show_cursor, coordinate_only, global_coordinates). The description adds only the high-level distinction between element-index and coordinate modes and the clipboard-restore behavior for paste, which is marginal value beyond the structured fields; baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('Perform an action on a macOS window or app') and then enumerates all eleven action values, so the agent knows exactly what surface the tool covers. It also draws the key internal distinction between element-index actions (background, no physical cursor movement) and coordinate actions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The line about element-index actions running in the background while coordinate actions move the physical cursor implicitly signals a preference for element_index, and the schema notes it comes from get_window_state. But there is no explicit when-to-use/when-not guidance, no prerequisites called out, and no direct routing to siblings such as get_window_state (to obtain an index) or cursor_control.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools