analyze_screen
Capture screenshots and analyze Android screen content to understand UI elements and structure for automation and testing purposes.
Instructions
Capture a screenshot and analyze the screen content. Returns both the visual screenshot and a text summary of the UI tree for comprehensive screen understanding. Use this when you need to understand the full screen context.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "analyze_screen"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| device_id | No | Device serial number |
Implementation Reference
- src/vision/visual-analyzer.ts:24-51 (handler)The logic for capturing the screenshot and UI tree to analyze the screen.
export async function analyzeScreen(deviceId?: string): Promise<ScreenAnalysis> { const resolved = await deviceManager.resolveDeviceId(deviceId); // Capture screenshot const screenshot = await captureScreenshot(resolved, { resize: { width: 1080, height: 1920 }, }); // Try to get UI tree summary (best effort) let uiSummary = ''; try { const tree = await getUITree(resolved); uiSummary = summarizeTree(tree); } catch (error) { log.warn('UI tree dump failed during analysis, proceeding with screenshot only', { error: error instanceof Error ? error.message : String(error), }); } const screenSize = await deviceManager.getScreenSize(resolved); return { screenshot, uiSummary, screenSize, deviceInfo: `Screen: ${screenSize.width}x${screenSize.height}`, }; } - src/controllers/vision-tools.ts:56-89 (registration)The MCP tool registration for the 'analyze_screen' tool.
server.registerTool( 'analyze_screen', { description: 'Capture a screenshot and analyze the screen content. Returns both the visual screenshot and a text summary of the UI tree for comprehensive screen understanding. Use this when you need to understand the full screen context.', inputSchema: { device_id: z.string().optional().describe('Device serial number'), }, }, async ({ device_id }) => { return await metrics.measure('analyze_screen', device_id || 'default', async () => { const analysis = await analyzeScreen(device_id); const content: Array<{ type: 'text'; text: string } | { type: 'image'; data: string; mimeType: string }> = []; content.push({ type: 'image' as const, data: analysis.screenshot.base64, mimeType: 'image/png', }); content.push({ type: 'text' as const, text: JSON.stringify({ success: true, screenSize: analysis.screenSize, deviceInfo: analysis.deviceInfo, uiSummary: analysis.uiSummary, }, null, 2), }); return { content }; }); } );