mobile_click_on_screen_at_coordinates
Simulate screen clicks at specific x,y coordinates for mobile automation tasks on iOS and Android devices. Use coordinates or accessibility snapshots for precise interactions.
Instructions
Click on the screen at given x,y coordinates. If clicking on an element, use the list_elements_on_screen tool to find the coordinates.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | The x coordinate to click on the screen, in pixels | |
| y | Yes | The y coordinate to click on the screen, in pixels |
Implementation Reference
- src/server.ts:332-345 (registration)Registration of the 'mobile_click_on_screen_at_coordinates' tool, including input schema (device, x, y) and handler that delegates to Robot.tap(x, y) via getRobotFromDevicetool( "mobile_click_on_screen_at_coordinates", "Click on the screen at given x,y coordinates. If clicking on an element, use the list_elements_on_screen tool to find the coordinates.", { device: z.string().describe("The device identifier to use. Use mobile_list_available_devices to find which devices are available to you."), x: z.number().describe("The x coordinate to click on the screen, in pixels"), y: z.number().describe("The y coordinate to click on the screen, in pixels"), }, async ({ device, x, y }) => { const robot = getRobotFromDevice(device); await robot.tap(x, y); return `Clicked on screen at coordinates: ${x}, ${y}`; } );
- src/android.ts:437-439 (handler)AndroidRobot.tap implementation: executes 'adb shell input tap x y'public async tap(x: number, y: number): Promise<void> { this.adb("shell", "input", "tap", `${x}`, `${y}`); }
- src/webdriver-agent.ts:149-174 (handler)WebDriverAgent.tap implementation for iOS: sends W3C pointer actions (move to (x,y), down, pause 100ms, up) via /actions endpoint. Used by both IosRobot and Simctlpublic async tap(x: number, y: number) { await this.withinSession(async sessionUrl => { const url = `${sessionUrl}/actions`; await fetch(url, { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ actions: [ { type: "pointer", id: "finger1", parameters: { pointerType: "touch" }, actions: [ { type: "pointerMove", duration: 0, x, y }, { type: "pointerDown", button: 0 }, { type: "pause", duration: 100 }, { type: "pointerUp", button: 0 } ] } ] }), }); }); }
- src/server.ts:153-179 (helper)getRobotFromDevice helper: dispatches to platform-specific Robot (AndroidRobot, IosRobot, or Simctl) based on device IDconst getRobotFromDevice = (device: string): Robot => { const iosManager = new IosManager(); const androidManager = new AndroidDeviceManager(); const simulators = simulatorManager.listBootedSimulators(); const androidDevices = androidManager.getConnectedDevices(); const iosDevices = iosManager.listDevices(); // Check if it's a simulator const simulator = simulators.find(s => s.name === device); if (simulator) { return simulatorManager.getSimulator(device); } // Check if it's an Android device const androidDevice = androidDevices.find(d => d.deviceId === device); if (androidDevice) { return new AndroidRobot(device); } // Check if it's an iOS device const iosDevice = iosDevices.find(d => d.deviceId === device); if (iosDevice) { return new IosRobot(device); } throw new ActionableError(`Device "${device}" not found. Use the mobile_list_available_devices tool to see available devices.`); };