observe
Capture the view hierarchy of displayed elements on Android or iOS screens for mobile automation and testing purposes.
Instructions
Get the view hierarchy of what is displayed on screen
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| platform | Yes | Target platform |
Implementation Reference
- src/server/observeTools.ts:47-52 (registration)Registers the 'observe' tool using ToolRegistry.registerDeviceAware with name, description, input schema, and handler function.ToolRegistry.registerDeviceAware( "observe", "Get the view hierarchy of what is displayed on screen", observeSchema, observeHandler, );
- src/server/observeTools.ts:21-29 (handler)Handler function that executes the 'observe' tool: instantiates ObserveScreen with the device and calls its execute() method, wraps in JSON response, handles errors.const observeHandler = async (device: BootedDevice) => { try { const observeScreen = new ObserveScreen(device); const result = await observeScreen.execute(); return createJSONToolResponse(result); } catch (error) { throw new ActionableError(`Failed to execute observe: ${error}`); } };
- src/server/observeTools.ts:10-12 (schema)Zod input schema for the 'observe' tool, validating the required 'platform' parameter.export const observeSchema = z.object({ platform: z.enum(["android", "ios"]).describe("Target platform") });
- Core execute() method in ObserveScreen class called by the tool handler. Orchestrates collection of screen size, system insets, rotation, view hierarchy, active window, screenshot, caching results, with platform-specific (android/ios) logic and parallel execution.async execute(queryOptions?: ViewHierarchyQueryOptions): Promise<ObserveResult> { try { logger.debug("Executing observe command"); const startTime = Date.now(); // Create base result object with timestamp const result = this.createBaseResult(); // Collect all data components with parallelization await this.collectAllData(result, queryOptions); // Cache the result for future use await this.cacheObserveResult(result); logger.debug("Observe command completed"); logger.debug(`Total observe command execution took ${Date.now() - startTime}ms`); return result; } catch (err) { logger.error("Critical error in observe command:", err); return { timestamp: new Date().toISOString(), screenSize: { width: 0, height: 0 }, systemInsets: { top: 0, right: 0, bottom: 0, left: 0 }, error: "Observation failed due to device access error" }; } }