pointer_click
Click at a screen position and verify the outcome: detects whether the click landed on a UI element or dead space, shows the affected window only when something changes, and can refuse the click if another window occludes the target.
Instructions
Click at an absolute screen position. Reports whether it LANDED on anything -- the screen is compared before and after, so a click into dead space says so instead of looking exactly like one that worked -- and shows you the result without a separate screenshot. Also reports whether the keyboard moved as a result. PASS expect_window: the click is refused if something else is under that point, which is the difference between a missed click and a click in someone else's window. Needs no consent dialog, unlike xdotool.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | ||
| y | No | ||
| ref | No | A widget number from the last screen_map -- the click lands at that widget's CURRENT position after its identity is re-verified, no coordinates needed. Give ref OR x/y, never both. Refs die at the next screen_map call. | |
| look | No | What to show you afterwards. Default "auto": wait for the screen to stop changing, measure how much this action changed, and attach a picture of the affected window only if something did change -- so a click that hit nothing costs no tokens and says so. "window" always attaches it, "screen" uses the whole desktop (slower, 6x the tokens), "region" uses look_at, false skips all of it. Use false for the middle of a sequence you are going to check at the end anyway. | auto |
| count | No | 2 for a double click. | |
| button | No | left | |
| look_at | No | Rectangle for look:"region", in screen pixels. Object form {x, y, width, height} or array form [x, y, width, height]. | |
| hover_first | No | Approach the point and settle before clicking, so a toolkit that only arms a button on hover gets its motion event. Chromium/CEF/Electron buttons (Creative Cloud, Spotify, 'desktop web' apps) commonly ignore a bare click and report nothing changed. Costs ~0.25s. | |
| on_occluded | No | What to do when expect_window is not the window at that point. Default refuses and names the blocker with its id and geometry. "click_topmost" clicks whatever is in front instead, in this same call, and says which window received it -- for a dialog that spawned over the button you were aiming at. | refuse |
| settle_max_s | No | How long to wait for the screen to stop changing before looking. Raise it for an app that animates slowly; set it to 0 to capture immediately. | |
| expect_window | No | The window this click is aimed at (id, wm_class or title fragment). Nothing is clicked if it is not the window at that point. |