get_ui_elements_info

Instructions

Implementation Reference

• puppeteer.py:560-607 (handler)

  The main tool handler decorated with @mcp.tool(). It calls the get_ui_elements helper to fetch interactive UI elements and formats them into a standardized dictionary response with success status, elements list, and error handling.

  @mcp.tool()
  async def get_ui_elements_info(device_id: str = None) -> dict:
      """Get detailed information about all interactive UI elements on the screen including their coordinates and properties."""
      try:
          elements = get_ui_elements(device_id)
  
          elements_info = []
          for i, element in enumerate(elements):
              elements_info.append({
                  "index": i,
                  "name": element.name,
                  "center_coordinates": {
                      "x": element.coordinates.x,
                      "y": element.coordinates.y
                  },
                  "bounding_box": {
                      "x1": element.bounding_box.x1,
                      "y1": element.bounding_box.y1,
                      "x2": element.bounding_box.x2,
                      "y2": element.bounding_box.y2
                  },
                  "class_name": element.class_name,
                  "clickable": element.clickable,
                  "focusable": element.focusable
              })
  
          return {
              "success": True,
              "message": f"Found {len(elements)} interactive UI elements",
              "device_id": device_id or "default",
              "elements": elements_info,
              "count": len(elements)
          }
  
      except ConnectionError as e:
          return {
              "success": False,
              "error": f"Device connection failed: {e}",
              "elements": [],
              "count": 0
          }
      except Exception as e:
          return {
              "success": False,
              "error": f"Failed to get UI elements: {e}",
              "elements": [],
              "count": 0
          }

• puppeteer.py:127-165 (helper)

  Key helper function that parses the Android UI hierarchy XML dump from uiautomator2, identifies interactive, visible, and enabled elements, extracts their bounds, names, and properties, and returns a list of ElementNode objects.

  def get_ui_elements(device_id: str = None) -> list[ElementNode]:
      """Get interactive UI elements from the device"""
      try:
          device = get_device_connection(device_id)
  
          # Get UI hierarchy XML
          tree_string = device.dump_hierarchy()
          element_tree = ElementTree.fromstring(tree_string)
  
          interactive_elements = []
          nodes = element_tree.findall('.//node[@visible-to-user="true"][@enabled="true"]')
  
          for node in nodes:
              if is_interactive(node):
                  coords = extract_coordinates(node)
                  if not coords:
                      continue
  
                  x1, y1, x2, y2 = coords
                  name = get_element_name(node)
  
                  if not name:
                      continue
  
                  x_center, y_center = get_center_coordinates((x1, y1, x2, y2))
  
                  interactive_elements.append(ElementNode(
                      name=name,
                      coordinates=CenterCord(x=x_center, y=y_center),
                      bounding_box=BoundingBox(x1=x1, y1=y1, x2=x2, y2=y2),
                      class_name=node.get('class', ''),
                      clickable=node.get('clickable') == 'true',
                      focusable=node.get('focusable') == 'true'
                  ))
  
          return interactive_elements
      except Exception as e:
          raise RuntimeError(f"Failed to get UI elements: {e}")

• puppeteer.py:45-53 (helper)

  Dataclass defining the structure for UI element information used by get_ui_elements and the main handler.

  @dataclass
  class ElementNode:
      name: str
      coordinates: CenterCord
      bounding_box: BoundingBox
      class_name: str = ""
      clickable: bool = False
      focusable: bool = False

• puppeteer.py:105-111 (helper)

  Helper function to determine if a UI node is interactive based on attributes or class name matching INTERACTIVE_CLASSES.

  def is_interactive(node) -> bool:
      """Check if a UI element is interactive"""
      attributes = node.attrib
      return (attributes.get('focusable') == "true" or
              attributes.get('clickable') == "true" or
              attributes.get('class') in INTERACTIVE_CLASSES)

• puppeteer.py:560-560 (registration)

  The @mcp.tool() decorator registers the get_ui_elements_info function with the FastMCP server.

  @mcp.tool()