Skip to main content
Glama

ui_act

Destructive

Act on UI element refs to click, type, select, scroll, drag, or press keys in Windows apps without moving the mouse or taking user focus.

Instructions

Act on a ref from the latest ui_tree/ui_wait/ui_ocr without moving the mouse or the user's focus. click | type (replace text) | check | uncheck | select (value= inside a list/combo) | expand | collapse | scroll | right-click | double-click | hover | wheel (value 'down 3') | drag (value= or 'dx,dy') | press (value 'Enter', 'ctrl+a', 'Down Down Enter'). press works in the background on classic controls and on a visible text field of a browser or Electron window. focus, and press on other WPF/browser controls (except Enter on a button or link), take the user's foreground: only with allow_foreground after the user agreed; press hands the foreground back once the keys are in, focus keeps it. If an action fails, never fall back to moving the real mouse or bringing the window forward yourself; tell the user. Refs expire when a window is read again.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "ref": "<ref>",
      "action": "<action>"
    },
    "name": "ui_act"
  }
}

Input Schema

NameRequiredDescriptionDefault
refYese.g. e12, or t3 from ui_ocr (click, double-click, right-click, hover, wheel, drag)
valueNotext for type, item for select, keys for press, target ref (e or t) or 'dx,dy' for drag, direction for wheel
actionYes
allow_foregroundNopermit taking the user's foreground window

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (destructiveHint=true, readOnlyHint=false), the description discloses non-obvious behavior: refs expire on re-read, press runs in the background on classic controls and visible browser/Electron text fields, focus and certain press targets steal the user's foreground, press returns it afterward, and failures must be surfaced rather than retried via real input.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The critical constraint (background, no mouse/focus movement) is front-loaded and every sentence carries operational content, but the action list and foreground caveats form one dense run-on passage that is harder to scan than it could be.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a destructive, foreground-sensitive input tool with no output schema, the description covers safety, foreground policy, failure handling, and ref lifetime. It does not say what a successful or failed call returns or how an agent should verify the UI changed, which would help close the loop after destructive actions.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 75%, but the description adds substantial meaning beyond it: per-action value syntax such as 'down 3' for wheel, 'dx,dy' or a target ref for drag, comma-separated key chords ('ctrl+a', 'Down Down Enter'), and the fact that refs may be e-codes from ui_tree or t-codes from ui_ocr.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb (act on a ref) and enumerates the full action vocabulary (click, type, check, drag, press, etc.), and it scopes the ref source to the latest ui_tree/ui_wait/ui_ocr. This clearly distinguishes it from screenshot/coordinate siblings like capture, crop, or ui_ocr.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It states when to use it (refs from the most recent ui_tree/ui_wait/ui_ocr), when a foreground-taking action is acceptable (only with allow_foreground after the user agreed), and explicitly forbids a fallback path (never move the real mouse or bring the window forward). This is close to a complete routing and constraint statement.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.