describe_ui
Retrieve the full view hierarchy, including precise frame coordinates (x, y, width, height) for all visible elements in a simulator. Use JSON tree output for accurate UI automation and layout validation.
Instructions
Gets entire view hierarchy with precise frame coordinates (x, y, width, height) for all visible elements. Use this before UI interactions or after layout changes - do NOT guess coordinates from screenshots. Returns JSON tree with frame data for accurate automation.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| simulatorUuid | Yes |
Implementation Reference
- Core handler logic for the describe_ui tool: executes 'axe describe-ui' command on the simulator to fetch UI hierarchy with precise frame coordinates, handles errors, and formats response with next steps.export async function describe_uiLogic( params: DescribeUiParams, executor: CommandExecutor, axeHelpers: AxeHelpers = { getAxePath, getBundledAxeEnvironment, createAxeNotAvailableResponse, }, ): Promise<ToolResponse> { const toolName = 'describe_ui'; const { simulatorId } = params; const commandArgs = ['describe-ui']; log('info', `${LOG_PREFIX}/${toolName}: Starting for ${simulatorId}`); try { const responseText = await executeAxeCommand( commandArgs, simulatorId, 'describe-ui', executor, axeHelpers, ); // Record the describe_ui call for warning system recordDescribeUICall(simulatorId); log('info', `${LOG_PREFIX}/${toolName}: Success for ${simulatorId}`); return { content: [ { type: 'text', text: 'Accessibility hierarchy retrieved successfully:\n```json\n' + responseText + '\n```', }, { type: 'text', text: `Next Steps: - Use frame coordinates for tap/swipe (center: x+width/2, y+height/2) - Re-run describe_ui after layout changes - Screenshots are for visual verification only`, }, ], }; } catch (error) { log('error', `${LOG_PREFIX}/${toolName}: Failed - ${error}`); if (error instanceof DependencyError) { return axeHelpers.createAxeNotAvailableResponse(); } else if (error instanceof AxeError) { return createErrorResponse( `Failed to get accessibility hierarchy: ${error.message}`, error.axeOutput, ); } else if (error instanceof SystemError) { return createErrorResponse( `System error executing axe: ${error.message}`, error.originalError?.stack, ); } return createErrorResponse( `An unexpected error occurred: ${error instanceof Error ? error.message : String(error)}`, ); } }
- Zod schema and TypeScript type definition for describe_ui tool input parameters (simulatorId: UUID string).const describeUiSchema = z.object({ simulatorId: z.string().uuid('Invalid Simulator UUID format'), }); // Use z.infer for type safety type DescribeUiParams = z.infer<typeof describeUiSchema>;
- src/mcp/tools/ui-testing/describe_ui.ts:111-127 (registration)MCP tool definition and registration: exports default object with name 'describe_ui', description, schema (without simulatorId for public), and session-aware handler wrapping describe_uiLogic.export default { name: 'describe_ui', description: 'Gets entire view hierarchy with precise frame coordinates (x, y, width, height) for all visible elements. Use this before UI interactions or after layout changes - do NOT guess coordinates from screenshots. Returns JSON tree with frame data for accurate automation.', schema: publicSchemaObject.shape, // MCP SDK compatibility handler: createSessionAwareTool<DescribeUiParams>({ internalSchema: describeUiSchema as unknown as z.ZodType<DescribeUiParams>, logicFunction: (params: DescribeUiParams, executor: CommandExecutor) => describe_uiLogic(params, executor, { getAxePath, getBundledAxeEnvironment, createAxeNotAvailableResponse, }), getExecutor: getDefaultCommandExecutor, requirements: [{ allOf: ['simulatorId'], message: 'simulatorId is required' }], }), };
- Supporting helper function that executes axe CLI commands, manages binary detection, UDID injection, environment vars, and comprehensive error wrapping (DependencyError, AxeError, SystemError).async function executeAxeCommand( commandArgs: string[], simulatorId: string, commandName: string, executor: CommandExecutor = getDefaultCommandExecutor(), axeHelpers: AxeHelpers = { getAxePath, getBundledAxeEnvironment, createAxeNotAvailableResponse }, ): Promise<string> { // Get the appropriate axe binary path const axeBinary = axeHelpers.getAxePath(); if (!axeBinary) { throw new DependencyError('AXe binary not found'); } // Add --udid parameter to all commands const fullArgs = [...commandArgs, '--udid', simulatorId]; // Construct the full command array with the axe binary as the first element const fullCommand = [axeBinary, ...fullArgs]; try { // Determine environment variables for bundled AXe const axeEnv = axeBinary !== 'axe' ? axeHelpers.getBundledAxeEnvironment() : undefined; const result = await executor(fullCommand, `${LOG_PREFIX}: ${commandName}`, false, axeEnv); if (!result.success) { throw new AxeError( `axe command '${commandName}' failed.`, commandName, result.error ?? result.output, simulatorId, ); } // Check for stderr output in successful commands if (result.error) { log( 'warn', `${LOG_PREFIX}: Command '${commandName}' produced stderr output but exited successfully. Output: ${result.error}`, ); } return result.output.trim(); } catch (error) { if (error instanceof Error) { if (error instanceof AxeError) { throw error; } // Otherwise wrap it in a SystemError throw new SystemError(`Failed to execute axe command: ${error.message}`, error); } // For any other type of error throw new SystemError(`Failed to execute axe command: ${String(error)}`); } }