mobile_list_elements_on_screen
Identify on-screen elements and their coordinates for iOS and Android applications, providing display text or accessibility labels without caching results. Essential for mobile automation and accessibility testing.
Instructions
List elements on screen and their coordinates, with display text or accessibility label. Do not cache this result.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| noParams | Yes |
Implementation Reference
- src/server.ts:277-312 (registration)Registration of the 'mobile_list_elements_on_screen' tool using the 'tool' helper function. Includes the input schema (empty) and the complete inline handler implementation that fetches screen elements via robot.getElementsOnScreen(), processes them, and returns a JSON-formatted list.tool( "mobile_list_elements_on_screen", "List elements on screen and their coordinates, with display text or accessibility label. Do not cache this result.", { noParams }, async ({}) => { requireRobot(); const elements = await robot!.getElementsOnScreen(); const result = elements.map(element => { const out: any = { type: element.type, text: element.text, label: element.label, name: element.name, value: element.value, identifier: element.identifier, coordinates: { x: element.rect.x, y: element.rect.y, width: element.rect.width, height: element.rect.height, }, }; if (element.focused) { out.focused = true; } return out; }); return `Found these elements on screen: ${JSON.stringify(result)}`; } );
- src/server.ts:283-311 (handler)The handler function for the tool. Ensures a robot/device is selected, retrieves UI elements from the screen using robot.getElementsOnScreen(), maps them to a simplified object with type, text, label, etc., and coordinates, and returns a descriptive string with JSON data.async ({}) => { requireRobot(); const elements = await robot!.getElementsOnScreen(); const result = elements.map(element => { const out: any = { type: element.type, text: element.text, label: element.label, name: element.name, value: element.value, identifier: element.identifier, coordinates: { x: element.rect.x, y: element.rect.y, width: element.rect.width, height: element.rect.height, }, }; if (element.focused) { out.focused = true; } return out; }); return `Found these elements on screen: ${JSON.stringify(result)}`; }
- src/server.ts:280-282 (schema)Input schema for the tool: no parameters required (z.object({})).{ noParams },
- src/server.ts:49-50 (helper)Helper schema definition for tools with no input parameters.// an empty object to satisfy windsurf const noParams = z.object({});
- src/server.ts:52-78 (helper)Helper function 'tool' used to register all MCP tools, including error handling, logging, and wrapping the custom handler into MCP CallToolResult format.const tool = (name: string, description: string, paramsSchema: ZodRawShape, cb: (args: z.objectOutputType<ZodRawShape, ZodTypeAny>) => Promise<string>) => { const wrappedCb = async (args: ZodRawShape): Promise<CallToolResult> => { try { trace(`Invoking ${name} with args: ${JSON.stringify(args)}`); const response = await cb(args); trace(`=> ${response}`); return { content: [{ type: "text", text: response }], }; } catch (error: any) { if (error instanceof ActionableError) { return { content: [{ type: "text", text: `${error.message}. Please fix the issue and try again.` }], }; } else { // a real exception trace(`Tool '${description}' failed: ${error.message} stack: ${error.stack}`); return { content: [{ type: "text", text: `Error: ${error.message}` }], isError: true, }; } } }; server.tool(name, description, paramsSchema, args => wrappedCb(args)); };