observe
Retrieve the view hierarchy displayed on screen for mobile automation testing. Supports Android and iOS platforms to analyze UI structure.
Instructions
Get the view hierarchy of what is displayed on screen
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| platform | Yes | Target platform |
Implementation Reference
- src/server/observeTools.ts:21-29 (handler)Handler function that executes the observe tool by creating an ObserveScreen instance and calling its execute method, returning the JSON response.const observeHandler = async (device: BootedDevice) => { try { const observeScreen = new ObserveScreen(device); const result = await observeScreen.execute(); return createJSONToolResponse(result); } catch (error) { throw new ActionableError(`Failed to execute observe: ${error}`); } };
- src/server/observeTools.ts:10-12 (schema)Input schema (Zod) for the observe tool, specifying the target platform.export const observeSchema = z.object({ platform: z.enum(["android", "ios"]).describe("Target platform") });
- src/server/observeTools.ts:47-52 (registration)Registers the observe tool with the ToolRegistry, specifying name, description, schema, and handler.ToolRegistry.registerDeviceAware( "observe", "Get the view hierarchy of what is displayed on screen", observeSchema, observeHandler, );
- src/models/ObserveResult.ts:10-52 (schema)Output type definition (TypeScript interface) for the result returned by the observe tool.export interface ObserveResult { /** * Timestamp when observation was made * Can be a number (milliseconds) or ISO string depending on context */ timestamp: string | number; /** Screen dimensions */ screenSize: ScreenSize; /** System UI insets */ systemInsets: SystemInsets; /** Screen rotation (0: portrait, 1: landscape 90°, 2: reverse portrait 180°, 3: reverse landscape 270°) */ rotation?: number; /** View hierarchy data */ viewHierarchy?: ViewHierarchyResult; /** Active window information */ activeWindow?: ActiveWindowInfo; /** * Categorized elements from the view hierarchy */ elements?: { clickable: Element[]; scrollable: Element[]; text: Element[]; }; /** * The single currently focused UI element from the view hierarchy * Contains the element that has focus state set to true */ focusedElement?: Element; /** Whether a system intent chooser dialog was detected */ intentChooserDetected?: boolean; /** Error message if observation failed partially or completely */ error?: string; }
- Core execution logic in ObserveScreen.execute(), which orchestrates collection of screen data, view hierarchy, caching, and returns ObserveResult.async execute(queryOptions?: ViewHierarchyQueryOptions): Promise<ObserveResult> { try { logger.debug("Executing observe command"); const startTime = Date.now(); // Create base result object with timestamp const result = this.createBaseResult(); // Collect all data components with parallelization await this.collectAllData(result, queryOptions); // Cache the result for future use await this.cacheObserveResult(result); logger.debug("Observe command completed"); logger.debug(`Total observe command execution took ${Date.now() - startTime}ms`); return result; } catch (err) { logger.error("Critical error in observe command:", err); return { timestamp: new Date().toISOString(), screenSize: { width: 0, height: 0 }, systemInsets: { top: 0, right: 0, bottom: 0, left: 0 }, error: "Observation failed due to device access error" }; } }
- src/server/index.ts:27-27 (registration)Top-level call to registerObserveTools() during MCP server setup, which triggers observe tool registration.registerObserveTools();