get_ui_context
Capture current UI state with screenshots and element properties to analyze mobile app interfaces for testing and debugging purposes.
Instructions
Capture the current UI state including screenshot and interactive elements. Returns a compressed screenshot and a list of UI elements with their properties.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| platform | Yes | Target platform | |
| deviceId | No | Device ID or name (optional, uses first running device if not specified) | |
| includeAllElements | No | Include all elements, not just interactive ones (default: false) | |
| maxDepth | No | Maximum depth to traverse in UI hierarchy (default: 20) | |
| screenshotQuality | No | Screenshot JPEG quality 1-100 (default: 50) | |
| skipScreenshot | No | Skip screenshot capture for faster response (default: false) | |
| elementTypes | No | Filter to specific element types |
Implementation Reference
- src/tools/ui/get-ui-context.ts:38-79 (handler)Main handler function for the get_ui_context tool. Validates input arguments, prepares options, and delegates to platform-specific UI context capture functions (Android or iOS).export async function getUIContext(args: GetUIContextArgs): Promise<UIContext> { const { platform, deviceId, includeAllElements = false, maxDepth = 20, screenshotQuality = 50, skipScreenshot = false, elementTypes, } = args; // Validate platform if (!isPlatform(platform)) { throw Errors.invalidArguments(`Invalid platform: ${platform}. Must be 'android' or 'ios'`); } // Validate element types if provided let validElementTypes: ElementType[] | undefined; if (elementTypes && elementTypes.length > 0) { validElementTypes = []; for (const type of elementTypes) { if (ELEMENT_TYPES.includes(type as ElementType)) { validElementTypes.push(type as ElementType); } } } const options: UIContextOptions = { deviceId, includeAllElements, maxDepth, screenshotQuality, skipScreenshot, elementTypes: validElementTypes, }; if (platform === 'android') { return captureAndroidContext(options); } else { return captureIOSContext(options); } }
- src/tools/ui/get-ui-context.ts:18-33 (schema)TypeScript interface defining the input arguments schema for the get_ui_context tool.export interface GetUIContextArgs { /** Target platform */ platform: string; /** Target device ID or name */ deviceId?: string; /** Include all elements (not just interactive) */ includeAllElements?: boolean; /** Maximum depth to traverse in hierarchy */ maxDepth?: number; /** Screenshot quality (1-100) */ screenshotQuality?: number; /** Skip screenshot capture */ skipScreenshot?: boolean; /** Filter to specific element types */ elementTypes?: string[]; }
- src/tools/ui/get-ui-context.ts:110-157 (registration)Registration function for the get_ui_context tool, including tool description, detailed JSON input schema, and handler binding.export function registerGetUIContextTool(): void { getToolRegistry().register( 'get_ui_context', { description: 'Capture the current UI state including screenshot and interactive elements. Returns a compressed screenshot and a list of UI elements with their properties.', inputSchema: createInputSchema( { platform: { type: 'string', enum: ['android', 'ios'], description: 'Target platform', }, deviceId: { type: 'string', description: 'Device ID or name (optional, uses first running device if not specified)', }, includeAllElements: { type: 'boolean', description: 'Include all elements, not just interactive ones (default: false)', }, maxDepth: { type: 'number', description: 'Maximum depth to traverse in UI hierarchy (default: 20)', }, screenshotQuality: { type: 'number', description: 'Screenshot JPEG quality 1-100 (default: 50)', }, skipScreenshot: { type: 'boolean', description: 'Skip screenshot capture for faster response (default: false)', }, elementTypes: { type: 'array', items: { type: 'string', enum: ['button', 'text', 'input', 'image', 'list', 'scroll', 'container', 'switch', 'checkbox', 'other'], }, description: 'Filter to specific element types', }, }, ['platform'] ), }, (args) => getUIContext(args as unknown as GetUIContextArgs) ); }
- src/tools/register.ts:131-134 (registration)Invocation of the tool registration during the overall registerAllTools process.const { registerGetUIContextTool } = await import('./ui/get-ui-context.js'); const { registerInteractWithUITool } = await import('./ui/interact-with-ui.js'); registerGetUIContextTool();