Android Puppeteer

Instructions

Implementation Reference

• puppeteer.py:560-607 (handler)

  The main tool handler decorated with @mcp.tool(). It calls the get_ui_elements helper to fetch interactive UI elements and formats them into a standardized dictionary response with success status, elements list, and error handling.
  @mcp.tool() async def get_ui_elements_info(device_id: str = None) -> dict: """Get detailed information about all interactive UI elements on the screen including their coordinates and properties.""" try: elements = get_ui_elements(device_id) elements_info = [] for i, element in enumerate(elements): elements_info.append({ "index": i, "name": element.name, "center_coordinates": { "x": element.coordinates.x, "y": element.coordinates.y }, "bounding_box": { "x1": element.bounding_box.x1, "y1": element.bounding_box.y1, "x2": element.bounding_box.x2, "y2": element.bounding_box.y2 }, "class_name": element.class_name, "clickable": element.clickable, "focusable": element.focusable }) return { "success": True, "message": f"Found {len(elements)} interactive UI elements", "device_id": device_id or "default", "elements": elements_info, "count": len(elements) } except ConnectionError as e: return { "success": False, "error": f"Device connection failed: {e}", "elements": [], "count": 0 } except Exception as e: return { "success": False, "error": f"Failed to get UI elements: {e}", "elements": [], "count": 0 }
• puppeteer.py:127-165 (helper)

  Key helper function that parses the Android UI hierarchy XML dump from uiautomator2, identifies interactive, visible, and enabled elements, extracts their bounds, names, and properties, and returns a list of ElementNode objects.
  def get_ui_elements(device_id: str = None) -> list[ElementNode]: """Get interactive UI elements from the device""" try: device = get_device_connection(device_id) # Get UI hierarchy XML tree_string = device.dump_hierarchy() element_tree = ElementTree.fromstring(tree_string) interactive_elements = [] nodes = element_tree.findall('.//node[@visible-to-user="true"][@enabled="true"]') for node in nodes: if is_interactive(node): coords = extract_coordinates(node) if not coords: continue x1, y1, x2, y2 = coords name = get_element_name(node) if not name: continue x_center, y_center = get_center_coordinates((x1, y1, x2, y2)) interactive_elements.append(ElementNode( name=name, coordinates=CenterCord(x=x_center, y=y_center), bounding_box=BoundingBox(x1=x1, y1=y1, x2=x2, y2=y2), class_name=node.get('class', ''), clickable=node.get('clickable') == 'true', focusable=node.get('focusable') == 'true' )) return interactive_elements except Exception as e: raise RuntimeError(f"Failed to get UI elements: {e}")
• puppeteer.py:45-53 (helper)

  Dataclass defining the structure for UI element information used by get_ui_elements and the main handler.
  @dataclass class ElementNode: name: str coordinates: CenterCord bounding_box: BoundingBox class_name: str = "" clickable: bool = False focusable: bool = False
• puppeteer.py:105-111 (helper)

  Helper function to determine if a UI node is interactive based on attributes or class name matching INTERACTIVE_CLASSES.
  def is_interactive(node) -> bool: """Check if a UI element is interactive""" attributes = node.attrib return (attributes.get('focusable') == "true" or attributes.get('clickable') == "true" or attributes.get('class') in INTERACTIVE_CLASSES)
• puppeteer.py:560-560 (registration)

  The @mcp.tool() decorator registers the get_ui_elements_info function with the FastMCP server.
  @mcp.tool()