AutoMobile

Instructions

Implementation Reference

• src/server/observeTools.ts:21-29 (handler)

  Handler function that executes the observe tool by creating an ObserveScreen instance and calling its execute method, returning the JSON response.

  const observeHandler = async (device: BootedDevice) => {
    try {
      const observeScreen = new ObserveScreen(device);
      const result = await observeScreen.execute();
      return createJSONToolResponse(result);
    } catch (error) {
      throw new ActionableError(`Failed to execute observe: ${error}`);
    }
  };

• src/server/observeTools.ts:10-12 (schema)

  Input schema (Zod) for the observe tool, specifying the target platform.

  export const observeSchema = z.object({
    platform: z.enum(["android", "ios"]).describe("Target platform")
  });

• src/server/observeTools.ts:47-52 (registration)

  Registers the observe tool with the ToolRegistry, specifying name, description, schema, and handler.

  ToolRegistry.registerDeviceAware(
    "observe",
    "Get the view hierarchy of what is displayed on screen",
    observeSchema,
    observeHandler,
  );

• src/models/ObserveResult.ts:10-52 (schema)

  Output type definition (TypeScript interface) for the result returned by the observe tool.

  export interface ObserveResult {
    /**
     * Timestamp when observation was made
     * Can be a number (milliseconds) or ISO string depending on context
     */
    timestamp: string | number;
  
    /** Screen dimensions */
    screenSize: ScreenSize;
  
    /** System UI insets */
    systemInsets: SystemInsets;
  
    /** Screen rotation (0: portrait, 1: landscape 90°, 2: reverse portrait 180°, 3: reverse landscape 270°) */
    rotation?: number;
  
    /** View hierarchy data */
    viewHierarchy?: ViewHierarchyResult;
  
    /** Active window information */
    activeWindow?: ActiveWindowInfo;
  
    /**
     * Categorized elements from the view hierarchy
     */
    elements?: {
      clickable: Element[];
      scrollable: Element[];
      text: Element[];
    };
  
    /**
     * The single currently focused UI element from the view hierarchy
     * Contains the element that has focus state set to true
     */
    focusedElement?: Element;
  
    /** Whether a system intent chooser dialog was detected */
    intentChooserDetected?: boolean;
  
    /** Error message if observation failed partially or completely */
    error?: string;
  }

• src/features/observe/ObserveScreen.ts:485-511 (helper)

  Core execution logic in ObserveScreen.execute(), which orchestrates collection of screen data, view hierarchy, caching, and returns ObserveResult.

  async execute(queryOptions?: ViewHierarchyQueryOptions): Promise<ObserveResult> {
    try {
      logger.debug("Executing observe command");
      const startTime = Date.now();
  
      // Create base result object with timestamp
      const result = this.createBaseResult();
  
      // Collect all data components with parallelization
      await this.collectAllData(result, queryOptions);
  
      // Cache the result for future use
      await this.cacheObserveResult(result);
  
      logger.debug("Observe command completed");
      logger.debug(`Total observe command execution took ${Date.now() - startTime}ms`);
      return result;
    } catch (err) {
      logger.error("Critical error in observe command:", err);
      return {
        timestamp: new Date().toISOString(),
        screenSize: { width: 0, height: 0 },
        systemInsets: { top: 0, right: 0, bottom: 0, left: 0 },
        error: "Observation failed due to device access error"
      };
    }
  }

• src/server/index.ts:27-27 (registration)

  Top-level call to registerObserveTools() during MCP server setup, which triggers observe tool registration.

  registerObserveTools();