get_clickable_elements
Identify all interactive screen elements with coordinates to determine what can be tapped during Android UI testing and debugging.
Instructions
Get all clickable/interactive elements on screen with their coordinates. Perfect for understanding what can be tapped.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| device_serial | No |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |
Implementation Reference
- src/adb_mcp_server/server.py:259-307 (handler)The handler function decorated with @mcp.tool() that implements the get_clickable_elements tool. It fetches the UI hierarchy XML, parses for clickable nodes using regex, extracts properties like text, content-desc, resource-id, bounds (with calculated center and size), and class name. Returns a list of dictionaries for each clickable element.
@mcp.tool() def get_clickable_elements(device_serial: str | None = None) -> list[dict]: """ Get all clickable/interactive elements on screen with their coordinates. Perfect for understanding what can be tapped. """ xml = get_ui_hierarchy(device_serial) elements = [] # Parse clickable elements pattern = r'<node[^>]*clickable="true"[^>]*>' for match in re.finditer(pattern, xml): node = match.group() element = {} # Extract text text_match = re.search(r'text="([^"]*)"', node) if text_match: element['text'] = text_match.group(1) # Extract content-desc desc_match = re.search(r'content-desc="([^"]*)"', node) if desc_match: element['content_desc'] = desc_match.group(1) # Extract resource-id id_match = re.search(r'resource-id="([^"]*)"', node) if id_match: element['resource_id'] = id_match.group(1) # Extract bounds and calculate center bounds_match = re.search(r'bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"', node) if bounds_match: x1, y1 = int(bounds_match.group(1)), int(bounds_match.group(2)) x2, y2 = int(bounds_match.group(3)), int(bounds_match.group(4)) element['bounds'] = {'x1': x1, 'y1': y1, 'x2': x2, 'y2': y2} element['center'] = {'x': (x1 + x2) // 2, 'y': (y1 + y2) // 2} element['size'] = {'width': x2 - x1, 'height': y2 - y1} # Extract class class_match = re.search(r'class="([^"]*)"', node) if class_match: element['class'] = class_match.group(1) if element: elements.append(element) return elements - src/adb_mcp_server/server.py:247-257 (helper)Helper function get_ui_hierarchy used by get_clickable_elements to dump and retrieve the UI hierarchy XML from the device via uiautomator.
@mcp.tool() def get_ui_hierarchy(device_serial: str | None = None) -> str: """ Dump the complete UI hierarchy as XML. Shows all visible elements, their properties, bounds, and content descriptions. """ run_adb(["shell", "uiautomator", "dump", "/sdcard/ui_dump.xml"], device_serial) output = run_adb(["shell", "cat", "/sdcard/ui_dump.xml"], device_serial) run_adb(["shell", "rm", "/sdcard/ui_dump.xml"], device_serial) return output