locate_element
Find a UI element in an app window and return click coordinates. Uses local vision model for privacy; supports viewport-relative coordinates for Chrome.
Instructions
Find a UI element in an app window and return its click coordinates. Uses a local Ollama vision model — Claude never sees the image. For Google Chrome, pass viewport_bounds (computed from javascript_tool: {x: screenX, y: screenY + outerHeight - innerHeight, width: innerWidth, height: innerHeight}) to get viewport-relative coordinates usable directly with the Chrome computer tool. Without viewport_bounds, returns absolute screen coordinates.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | Ollama vision model to use (defaults to OLLAMA_MODEL env var, or gemma4:e4b) | gemma4:e4b |
| app_name | Yes | Exact macOS app name, e.g. 'Google Chrome', 'Safari', 'Terminal', 'Cursor' | |
| window_index | No | Which window to capture if the app has multiple (1 = frontmost) | |
| viewport_bounds | No | Screen-coordinate bounds of the region to capture (e.g. Chrome's content area). For Chrome: compute as {x: screenX, y: screenY + outerHeight - innerHeight, width: innerWidth, height: innerHeight} using window.screenX/Y/outerWidth/outerHeight/innerWidth/innerHeight from javascript_tool. When provided, returned coordinates are relative to this region's (0,0) — viewport-relative for Chrome. | |
| element_description | Yes | Natural language description of the element to find, e.g. 'the X close button in the top-right of the modal' |